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SUMMARY 

This  report  details  the  analysis  and  study  procedures  used  during  the  GAPCEEC  program. 

•  Section  I  —  Introduction  —  Summarizes  the  program  objectives,  back¬ 
ground,  and  reasons  for  conducting  the  program;  and,  provides  an  overview 
of  program  structure,  analysis  results,  and  conclusions. 

•  Section  II  —  Reliability-Growth  Model  Research  and  Selection  —  This 
section  discusses  the  literature  search,  dialogue  with  the  specialist,  initial 
criteria  for  comparing  models,  and  models  selected  for  evaluation. 

•  Section  III  —  Evaluation  of  Reliability-Growth  Models  and  Selection  of  the 
Best  Model  —  This  section  discusses  the  components  that  were  selected  for 
evaluation,  the  data  used,  the  methodology  used  to  evaluate  the  models,  and 
the  results  of  the  evaluation.  The  AMSAA/Duane  model  was  selected  as  the 
best  model. 

•  Section  IV  —  Program  Synthesis  —  This  section  discusses  the  confidence 
limits  for  the  AMSAA/Duane  model  and  the  Time  Series  model.  It  also  gives 
an  overview  of  the  procedure  (contained  in  Appendix  D  to  the  report)  for 
applying  the  AMSAA/Duane  model. 

•  Section  V  —  Warranty  Considerations  —  This  section  discusses  the 
application  of  the  AMSAA/Duane  reliability-growth  model  to  warranty 
evaluation. 

•  Section  VI  —  Results  and  Conclusions  —  This  section  summarizes  the 
significant  results  of  this  study. 

•  Appendix  A  —  List  of  Applicable  Documents  —  Presents  the  results  of  the 
literature  search  of  books,  articles,  and  reports  related  to  reliability-growth 
modeling. 

•  Appendix  B  —  Criteria  for  Reliability-Growth  Model  Selection  —  Criteria 
used  for  evaluating  the  selected  growth  models  for  determining  the  optimum 
model. 

•  Appendix  C  —  Maximum  Likelihood  Estimation  of  Parameters  of  Reliability- 
Growth  Models  —  Illustrates  the  mathematical  derivation  of  the  maximum 
likelihood  estimators  for  the  AMSAA  model,  Cox-Lewis  model,  and 
Modified  Duane  model. 

•  Appendix  D  —  Growth  Model  Application  Procedure  —  Contains  the 
procedure  for  applying  the  AMSAA/Duane  model  to  electronic  control 
development  programs. 

•  Appendix  E  —  Time  Series  Analysis  —  Contains  the  procedure  for  applying 
the  Box-Jenkins  Time  Series  Analysis  to  reliability-growth  situations. 


SECTION  i 


INTRODUCTION 


A.  BACKGROUND 

The  use  of  digital  electronic  controls  for  gas  turbine  engines  has  become  an  established  goal 
throughout  the  propulsion  industry.  Most  gas  turbine  engine  manufacturers,  in  conjunction  with 
Government  agencies,  are  evaluating  various  concepts  for  full  authority  electronic  control  of 
their  engines.  These  electronic  systems  will  be  required  to  surpass  the  reliability  demonstrated  by 
existing  hydromechanical  systems,  which  becomes  a  new  role  for  modern  electronics  in  that 
safety-of-flight  and  engine  operational  readiness  will  be  impacted  directly.  That  is,  engines  so 
equipped  will  be  safe  and  effective  to  operate  only  so  long  as  the  reliability  and  integrity  of  the 
electronics  remain  intact.  Ultimately,  the  entire  weapons  system  will  have  all  of  its  control 
systems  working  in  an  integrated  fashion  and  functioning  synergistically  rather  than  individual¬ 
ly.  High  mission  reliability  will  then  be  accomplished  through  redundancy  with  a  high  confidence 
factor  of  fail-operational  capability.  High  individual  component  reliabilities  will  be  absolutely 
necessary  to  accomplish  this  goal  within  acceptable  cost  and  maintainability  limits. 

There  is  probably  no  single  best  approach  to  high  reliability,  since  each  application  is  likely 
to  have  unique  requirements.  However,  reliability-growth  modeling  is  a  common  element  of 
various  approaches  which  can  be  useful  for  diverse  kinds  of  equipment  and  systems.  In  fact,  a 
significant  amount  of  information  accumulated  in  technical  literature  over  the  past  2  decades 
strongly  indicates  that  the  reliability-growth  modeling  concept  can  be  a  very  effective  tool  for  the 
assessment,  prediction  and  control  of  reliability  in  modern  electronic  equipments  and  systems. 

B.  PROBLEM 

The  ability  to  predict  the  reliability  of  a  control  system/component  population  within 
reasonably  close  limits,  at  any  point  in  its  development  and  operational  service  life,  is  a 
significant  requisite  to  achieving  the  very  high  reliability  goals  set  forth  for  future  engine  control 
systems.  At  any  point  in  time,  both  the  achieved  reliability  and  its  growth  trend  must  be 
identifiable  quantitatively  to  permit  timely  corrective  action  to  be  taken  when  it  appears  likely 
that  specified  reliability  goals  will  not  be  met  on  schedule. 

C.  OBJECTIVE 

Two  objectives  of  this  program  were  to:  (1)  select  the  optimum  reliability  growth  model  for 
application  to  future  electronic  engine  control  system  components  and  (2)  to  evolve  a  reliability- 
growth  modeling  analytical  procedure  which  would  provide  guidance  with  which  to  confidently 
assess,  predict,  and  control  the  reliability  growth  of  military  engine  control  systems  and 
components  throughout  their  development  and  service  lives.  In  addition,  an  assessment  was  to  be 
made  as  to  the  applicability  of  the  program  results  in  establishing  the  feasibility  and  validity  of 
applying  warranty  provisions  to  future  development  programs  for  military  aircraft  engines. 

D.  APPROACH 

A  technical  team  headed  by  Pratt  &  Whitney  (P&W),  including  team  members  from 
Hamilton  Standard  and  Chandler  Evans  Corporations,  was  assembled  to  pursue  this  objective, 
along  with  frequent  coordination  with  leaders  in  the  reliability-growth  modeling  field.  This  team 
included  personnel  with  extensive  backgrounds  in  control  system  design  and  integration, 
statistical  analysis  and  reliability  modeling. 


Guidance  required  to  analyze  and  predict  the  reliability  growth  which  can  be  expected  was 
obtained  by  reviewing  available  data  sources  and  reliability  analysis  tools.  Experience  with  the 
F100  EEC,  which  has  accumulated  over  1  million  operating  hours  in  the  United  States  Air  Force 
(USAF)  inventory,  provided  insight  into  many  areas  where  better  tools  for  reliability  analysis 
and  predictions  are  required;  including  insight  into  the  assets  and  liabilities  of  current  reliability 
data  tracking  systems.  These  systems  have  made  available  a  substantial  data  base  of  USAF, 
contractor,  and  vendor  reliability  data. 

Many  reliability  models  and  statistical  analysis  tools  are  available  which  may  be  used  to 
address  the  problems  of  predicting  and  assessing  the  reliability  growth  of  gas  turbine  control 
systems  and  components  with  varying  degrees  of  success. 

This  program  was  structured  to  review  the  literature  on  these  various  methods,  provide  a 
dialogue  with  experts  from  the  field,  and  evaluate  and  select  a  model  which  will  best  meet  the 
needs  of  the  control  components  of  the  future.  In  the  process  of  evaluating  modeling  techniques, 
due  regard  was  placed  on  the  sources  and  validity  of  the  data  sets,  accuracy  of  the  results,  and 
feasibility  of  the  models  to  evaluate  future  fault  tolerant  redundant  control  systems. 

E.  SYNOPSIS 

During  this  program,  more  than  20  models  of  4  major  classes,  along  with  more  than 
200  books,  articles,  and  reports  that  deal  with  reliability-growth  modeling,  were  selected  for 
review.  Dialogue  was  established  with  several  specialists  in  growth  modeling. 

Criteria  were  established,  involving  applicability,  flexibility,  ease  of  use,  and  short-and 
long-term  forecasting  precision,  for  model  comparisons.  Five  models  were  selected  for  further 
testing  with  data  bases  acquired  from  development  testing  and  field  service  of  components 
deployed  in  operational  service.  These  models  were: 

•  AMSAA  (Army  Material  Systems  Analysis  Activity) /Duane  Model 

•  Endless-Burn-in  Model 

•  Modified  Duane  Model 

•  Cox-Le-vis  Model 

•  Time  Series  Modeling. 

A  data  base  was  compiled  from  development  testing  and/or  field  service  for  six  major 
components,  representative  of  both  electronic  and  mechanical  control  systems.  Two  components, 
the  F100  JFC-90  Electronic  Engine  Control  (EEC)  and  F100  MFP-330  Main  Fuel  Pump  System, 
were  selected  for  use  as  a  basis  for  comparing  each  model’s  ability  to  forecast  reliability  over  the 
short-and  long-term. 

The  models  were  then  compared  via  the  evaluation  criteria,  and  the  AMSAA/Duane  Model 
was  selected  as  the  best  model.  The  Time  Series  and  Endless-Burn-In  models  were  also 
considered  acceptable.  The  best  method  for  estimating  the  AMSAA  model  parameters  which 
must  be  done  prior  to  model  application  was  found  to  be  the  method  of  maximum  likelihood. 

The  AMSAA/Duane  Model  and  Box-Jenkins  Time  Series  Modeling  were  reviewed  in  detail 
to  determine  how  to  use  confidence  intervals  for  estimates  of  reliability  and  goodness-of-fit  tests 
to  quantify  the  errors  in  assessment  and  forecasting  of  reliability.  Model  assumptions,  strengths 
and  weaknesses  were  also  reviewed  and  documented. 

A  procedure  was  then  drafted  to  assist  the  user  in  applying  reliability  growth  modeling 
techniques  to  advanced  control  system  development  programs.  Two  EEC’s,  the  P&W  Military 
F100  JFC-90  and  the  P&W  commercial  JT9D  EEC  103-2,  were  used  as  examples  in  the 
procedure. 


Assessment  of  the  applicability  of  growth  modeling  to  warranty  programs  revealed  that  care 
should  be  taken  when  using  the  models  to  assign  warranties  because  of  their  inadequacies  in 
making  long-term  projections.  However,  reliability-growth  modeling  does  appear  to  provide 
valuable  guidance  in  assessing  the  impact  of  warranty  programs  over  the  near  term. 
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SECTION  II 


RELIABILITY  GROWTH  MODEL  RESEARCH  AND  SELECTION 


A.  TASK  100  —  LITERATURE  SEARCH  AND  MODEL  SELECTION 
1.  Literature  Search 

The  United  Technologies’  library  svstem  was  used  in  conjunction  with  several  area 
university  libraries  (University  of  Florida,  Florida  State  University  and  Florida  Atlantic 
University),  a  computer  search  program  (DIALOG)  and  communication  with  several  specialists 
to  compile  a  bibliography  of  more  than  200  books,  articles  and  reports  related  to  reliability- 
growth  modeling.  This  bibliography,  shown  here  as  Appendix  A,  contains  literature  which  covers 
the  past  30  years  of  work  on  reliability-growth  assessment,  prediction  and  control.  This  literature 
has  been  grouped  by  model  type  and  subject  matter  for  easy  reference  as  shown  in  Table  1. 

TABLE  1.  LITERATURE  FROM  APPENDIX  A  CATEGORIZED  BY  MODEL 
TYPE  AND  CONTENT 


Subject 


Matter 

Methods 


Electronic 

Application 


Mechanical 

Application 


General 

Discussion 


Deterministic 

1,15,16, 

1,9,15,18 

23,24,25,38, 

9,133,177, 

4,16,18,21, 

17,18,19, 

23,24,25, 

46,133,142, 

190,209 

23,24,25,32, 

27,34,36, 

27,32,38, 

153,174 

34,37,40,41, 

41,54,55, 

40,41,53, 

42,46,46,53, 

56,58,64, 

64.68,73, 

57,62,63,65, 

73,84,115 

76,145, 

70,71,81,82, 

145,172, 

177,178, 

102,115,142, 

177,178, 

189,195. 

145.162,163, 

195,200, 

205,208, 

174,182,183, 

204,208, 

209,210 

185,197,203, 

210,212 

204,209,214 

Renewal  Process 

14,18.29. 

29,30,41, 

44 

30,44,59, 

59,175, 

175,176 

176 

Markov  Process 

2,3,14, 

2,14,26, 

2,78 

14,34,44 

18,26,34, 

78,175, 

44,59,66, 

175,176, 

184,188 

176 

Nonhomogeneous 

8,11,14. 

11.12,13, 

133 

11,12,13,48, 

8,10,11,13, 

Poisson  Process 

16,18,44, 

14,48,49, 

49,61,133 

16,44,45,48, 

48,49,50, 

50,51,52, 

60,67,145, 

51,02,61, 

61,71,72, 

186,197 

71,72,75, 

75,77,124 

77.124, 

145,149, 

145,149, 
164,195, 
196,21 1 

195,196 

Semi- Markov 

167,175, 

9,167,175 

9 

Process 

176,189 

176,189 

Time  Series 

3,5,33, 

5,33,39, 

191,192,193 

5,31,33 

Process 

74,83 

74,83,191 

192.193 

Bayesian 

69,83,88. 

69,83,88, 

104,105, 

134,165, 

134,165, 

194 

194 

40H2C 


a.  Dialogue  With  Specialists 


A  personal  dialogue  was  established  with  three  specialists  in  reliability-growth  modeling  as 
follows: 

Mr.  Alexander  G.  Bezat 

Honeywell,  Inc.  Avionics  Div. 

1625  Zarthan  Ave. 

St.  Louis  Park,  MN  55416 

Dr.  Larry  Crow 

US  Army  Material  System  —  Analysis  Activity 

Aberdeen  Proving  Ground 

Aberdeen,  MD  21005 

Dr.  Ronald  Scott 

United  Technologies  Inc. 

West  Palm  Beach,  FL  33402. 

Mr.  Alexander  Bezat,  a  reliability-growth  modeling  specialist  at  Honeywell  Inc.,  has 
successfully  applied  the  Endless-Burn- Ir.  (EBI)  model  to  commercial  aircraft  avionic  controls’ 
prog'-ams.  Mr.  Bezat  was  very  helpful  in  clarifying  the  assumptions  associated  with  the  EBI 
model  and  how  it  has  been  applied  at  Honeywell.  He  also  contributed  several  published  and 
unput  i  .shed  papers  on  application  of  the  EBI  model  as  noted  in  Appendix  A. 

Dr.  Larry  Crow,  a  reliability-growth  modeling  specialist  with  the  Army  Material  Systems 
Anali  sis  Agency  (AMSAA)  has  developed  and  successfully  applied  the  AMSAA/Duane  modeling 
procedure  and  was  responsible  for  the  development  of  MIL-HDBK-189  (Appendix  A),  the  most 
inclusive  and  widely  accepted  document  on  reliability-growth  modeling  presently  available. 
Dr.  Crow  was  helpful  in  clarifying  the  underlying  assumptions  of  the  AMSAA  model  and  how  to 
apply  it  to  a  wide  range  of  data.  He  also  recommended  methods  for  estimating  the  influences  of 
delayed  fixes  on  reliability.  He  also  made  available  several  papers  on  reliability-growth. 

Dr.  Ronald  Scott,  is  a  mathematician  with  United  Technologies  Research  Center  and  a 
specialist  in  Kalman  Filtering  and  Markov  Processes.  Dr.  Scott  was  helpful  in  clarifying  the 
requirements  for  applying  Markov  Process  models.  A  dialogue  was  also  established  with  several 
other  specialists  in  the  field  of  reliability  or  related  fields,  as  listed  in  Table  2. 

b.  Established  Criteria 

A  set  of  objectives  and  qualitative  criteria  was  established  to  judge  the  models’ 
effective. iess.  This  criteria,  as  listed  in  Appendix  R,  was  primarily  concerned  with  flexibility,  ease 
of  use,  short-  and  long-term  forecasting  precision,  and  statistical  validity.  A  list  of  statements 
describing  the  criteria  is  also  presented  in  Appendix  B.  These  statements  were  necessary  so  that 
qualitative  measures  of  goodness  could  be  assigned. 

2.  Models  Reviewed  and  Selected 

The  models  reviewed  fell  in<  o  one  or  more  of  the  following  general  categories:  Determinis¬ 
tic,  Poisson  Process,  Markov  Process/Time  Series,  Bayesian  Models. 


TABLE  2.  DIALOGUE 


DISCUSSIONS  WITH  SPECIALISTS 


1  Mr.  H.  E.  Ascher 
*2  Mr.  A.  Bezat 
*3  Dr.  L.  H.  Crow 

4  Dr.  K.  W.  Fertig 

5  Dr.  J.  M.  Finkelstein 

6  Dr.  P.  Holmes 

7  Dr.  W.  Jewell 

8  Dr.  P.  A.  W.  Lewis 

9  Dr.  W.  A.  Meeker 

10  Dr.  W.  Nelson 
H  Dr.  K.  Portier 
12  Dr.  F.  Proschan 
1.5  Dr.  R.  Scheaffer 

*14  Dr.  R.  Scott 
15  Dr.  N.  D  oingpunvalla 


•Indicates  personal  visits. 


Naval  Research  Lab,  Wash.,  DC 

Honeywell  Inc.,  Minn.,  MI 

U.S.  Arm?  Material  Systems,  Aberdeen,  MD 

Rockwell  Science  Center,  CA 

Hughes,  !nc.,  CA 

Clemaon  University,  SC 

University  of  California,  Berkeley,  CA 

Post  Graduate  Naval  School,  Monterey,  CA 

Iowa  State  University,  IA 

General  Electric,  Schenectady,  NY 

University  of  Florida,  FL 

Florida  State  University,  FL 

University  of  Florida,  FL 

United  Technologies  Corp.,  W.  Palm  Beach,  FL 

George  Washington  University,  Wash.,  DC 


NHPP/Serai  Markov  Process 
Deterministic  (Endless-Burn-In) 
NHPP  (AMSAA  Model) 

General  Comments 
NHPP  (Modified  Duane  Model) 
Markov  Processes/NHPP 
Bayesian  Models 
Time  Series 
General  Comments 
General  Comments 
Semi-Markov  Processes 
General  Comments 
Semi-Markov  Comments 
Markov  Processes 
Time  Series 


a.  Deterministic  Models 

The  form  of  Deterministic  models  (models  which  contain  no  random  elements)  is  assumed 
to  be  known.  That  is,  it  is  known  that  reliability  grows  via  the-  Deterministic  model.  The 
parameters  are  estimated  and  reliability  is  then  calculated  from  the  model.  Examples  of 
Deterministic  models  reviewed  in  Table  I  are  as  follows: 

•  The  Duane  Model: 

Cumulative  MTBF  =  y  (Cumulative  Time)“ 

•  The  Endless-Burn-In  Model: 

Instantaneous  failure  rate  =  K  (Average  Age)~“  +  A.R 

•  The  Gompertz  Model: 

Cumulative  failure  rate  -  K-AB'  (<'umulativ<; Time) 

•  The  Grumman  Model 

Instantaneous  failure  rate  =  A  e~K  (Cumulative  Time)  g 

b.  Poisson  Process  Models 

This  class  of  models  does  not  assume  that  the  form  of  the  reliability-growth  function  is 
known,  but  only  that  it  can  be  approximated  statistically.  Since  the  model  form  is  only  an 
approximation,  the  quality  of  fit  tests  are  necessary  to  determine  if  the  approximation  is 
reasonable.  The  Poisson  Process  model  assumes  that  events  occur  based  on  a  poisson  type  of 
distribution.  That  is,  the  probability  of  realizing  K  events  (failures)  by  some  test  time  t  is  as 
follows: 


P(x  =  k) 


[M(t)f  e-M(t) 
k! 
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Here  M(t)  is  the  mean  value  function.  If  M(t)  -  At,  the  process  is  a  Homogeneous  Poisson 
Process  (HPP)  and  the  time  between  failures  follow  an  exponential  distribution  (the  probability 
of  failure  by  time  t «  1  -  e-Xt). 

The  Poisson  Process  also  assumes  independent  increments.  Renewal  Theory  generalizes 
the  HPP  by  allowing  time  between  failures  to  have  other  distributions  besides  the  exponential.  A 
renewal  process  is  a  sequence  of  random  variables  [Yj,  Y2, . . .]  of  the  form  Y;  =  Xj  +  X,  +  . . .  + 
X;,  where  each  Xj  comes  from  a  common  distribution  F(X).  Therefore,  the  renewal  model 
assumes  that  each  repair  returns  the  system  to  good-as-new  state.  If  the  Poisson  Process  has  a 
more  general  mean  value  function  than  M(t)  =  At,  it  is  said  to  be  a  Nonhomogeneous  Poisson 
Process  (NHPP).  The  intensity  function,  p(t),  of  a  Poisson  Process  is  the  rate  at  which  failures 
are  occurring  and  is  related  to  the  mean  value  function  as  follows: 

M(t)  -  J‘p(x)dx. 


The  poisson  process  can  be  generalized  further  (Thompson)  to  a  general  point  process  but 
little  applied  work  has  been  done  at  that  level  of  abstraction.  The  NHPP  model  is  the  most 
popular  since  it  can  model  a  system  that  is  wearing  out.  Several  intensity  functions  have  been 
used  with  the  NHPP  model.  The  following  are  examples: 

•  The  AMSAA  model  used  by  Dr.  Crow, 

p(t)  =  Ap  (cumulative  test  time)  P"1; 

•  The  modified  Duane  Model  used  by  Dr.  Finkelstein, 

p(t)  =  AP  (cumulative  test  timejP"1  +  0; 

•  The  Cox-Lewis  model, 

p(t)  =  ga  +  Y  (cumulative  test  time). 

c.  Markov-Processes/Time  Sartos  Models 

H.  Akaike  (Appendix  A)  showed  that  Markovian  processes  and  the  autoregressive  moving 
average  process  were  related.  A  Markov  process  is  a  process  that  moves  from  state  i  to  state  j 
with  some  probability,  Pjj.  This  probability  is  independent  of  all  past  states,  and  is  dependent 
only  on  the  present  state.  The  time  spent  in  each  state  is  an  exponentially  distributed  random 
variable.  If  the  time  in  residence  is  not  exponential,  the  process  is  said  to  be  a  semi-Markov 
process.  Markovian  processes  usually  require  more  data,  to  estimate  the  probabilities,  than  is 
generally  available. 

Dr.  Singpurwalla  (Appendix  A)  suggests  that  time  series  methods  developed  by  Box  and 
Jenkins  be  used  to  model  reliability  growth.  The  Box-Jenkins  Autoregr  isive  Integrated  Moving 
Average  (ARIMA)  modeling  approach  is  attractive  in  that  no  specific  model  need  be  selected  in 
advance.  The  models  are  flexible  in  that  they  can  be  applied  to  a  wide  range  of  data  and  the 
methodology  has  a  built-in  theory  of  forecasting.  An  ARIMA  first  difference  model  could  be 
written  as  follows: 

Ac  -r  =  Ac  T_j  +  &r,  where  Ac  T  is  the  cumulative  failure  rate  at  time  T  and  &j>  is 

the  random  (normal  (0,  a))  error  at  time  T. 
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B.  SUMMARY  OF  TASK  100 


During  Task  100,  the  literature  was  reviewed  (Appendix  A),  the  criteria  established 
(Appendix  B)  and  five  models  were  recommended  for  carryover  into  Task  200.  These  models 
were: 


(1)  The  AMSAA/Duane  Model  (NHPP) 

(2)  The  Modified  Duane  Model  (NHPP) 

(3)  The  Cox-lewis  Model  (NHPP) 

(4)  The  Endless-Bum-In  Model  (Deterministic) 

(5)  Box -Jenkins  ARIMA  Models  (Time  Series  Process). 

These  models  were  representative  of  three  of  the  four  classes  of  models  considered.  Since  little 
information  was  available  on  Bayesian  models,  they  were  not  recommended  for  further  review. 
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SECTION  III 


EVALUATION  OF  RELIABILITY  GROWTH  MODELS 
ANO  THE  SELECTION  OF  THE  BEST  MODEMS) 

A.  DATA  COLLECTION  AND  REVIEW 

1.  S«4«ct*d  Components 

The  two  major  controls’  components  selected  for  use  in  this  study  were  the  F100  Electronic 
Engine  Control  (EEC)  and  the  F100  Main  Fuel  Pump  Systems  (MFPS)  shown  in  Figures  1  and 

2.  The  F100  EEC  is  an  engine-mounted,  digital,  electronic  supervisory  control.  The  F100  MFPS 
is  a  high  technology  vane  pump  and  variable  displacement  controller.  F100  components  currently 
have  over  3  million  hours  of  operational  experience  in  the  F-15  and  F-16  aircraft  These 
components  were  selected  because  of  the  quantity  and  quality  of  data  available  and  because  they 
represent  the  generic  types  of  components  that  are  applicable  to  future  engine  control  systems. 


FD  123982 


Figure  1.  F100  Electronic  Engine  Control 

In  order  to  expand  the  number  of  data  bases  and  number  of  different  type  components 
available  for  model  evaluation,  two  additional  F100  components  were  added:  These  were  the 
Compressor  Inlet  Variable  Vane  (CIVV)  Actuator  and  the  TT2.5  Sensor  shown  in  Figures  3  and 
4.  These  four  F100  controls  components  were  used  throughout  we  study  for  evaluating  and 
selecting  the  reliability  growth  model. 


E 


devious  page 
<S  BLANK 


£» 


11 


Servo  Return 


(Camber  CIVV) 


(Axial  CIVV) 
Aug  Pump 
Discharge 
Pressure 


Faflur* 


Body 
Prmur* 

(Pfcb> 

(from  Uniftod  Control) 


Temperature  Signal 
Praaaura  (P«)  lo 
Unified  Control 


Tj2.5  Tamporaturt 
Input 


FD  123974A 


Figure  4.  F100  Fan  Exit  Temperature  (TT2.5)  Sensor 


Two  additional  electronic  engine  controls  currently  in  development  were  also  included  in 
the  data  base  for  model  evaluation.  The  Digital  Electronic  Engine  Control  (DEEC)  developed  for 
the  F100  engine  has  completed  50,000  hours  of  Combined  Environment  Reliability  Testing 
(CERT).  CERT  is  an  effective  method  for  subjecting  a  component  to  the  combined  environment 
(vibration,  temperature,  pressure,  humidity)  that  the  component  would  experience  during 
operatiional  service.  Testing  under  these  controlled  conditions,  24-hours  per  day,  permits  the 
accumulation  of  many  hours  of  exposure  during  a  much  shorter  time  period  than  could  be 
accomplished  during  operational  service.  The  Electronic  Engine  Control  (EEC)  for  the 
commercial  JT9D  engine  has  also  completed  a  50,000-hour  CERT  and  has  accumulated  over 
300,000  hours  of  flight  test  experience  on  JT8D  engines.  These  programs  represent  current  state- 
of-the-art  electronic  controls  with  testing  designed  specifically  to  find  and  eliminate  problems. 

2.  Data  Collection 

a.  Ft  00  Controls  Development  Data 

Development  data  for  the  F100  controls’  components  was  extracted  from  the  P&W/GPD 
reporting  system.  Shown  in  Table  3  is  an  example  of  the  data  available  from  this  computerized 
system. 

This  data  system  includes  malfunctions  from  engine  development,  flight  test,  and 
production  testing  of  F100  components  covering  the  time  period  1969  to  1982.  All  data  for  the 
four  F100  controls’  components  was  reviewed  to  ensure  that  all  reported  malfunctions  were 
chargeable  to  the  component  and  not  a  result  of  test  stand  interface  or  instrumentation 
problems. 


TABLE  3.  F100  REPORTING  DATA 


FMR  No.:  DATE:  75/12/19  ENGINE:  P0264  TYPE:  JTF22  MODEL:  F  100(3)  RUN  NO.:  BUILD:  B  BASE: 

F9096 

APE:  COMPONENT:  CONTROL-EEC  PART  NO.:  4044335  SERIAL  NO.:  AESC0297  OCCURRED  FAIL  TYPE: 

AT: 


DESCRIPTION: 

NOC1VV  SCHEDULE 


ANALYSIS: 

INTERMITTENT  SOLDER  JOINT  AT  TB303-12 
BETWEEN  FLEXTAPE  AND  PIN. 

CORRECTIVE  ACTION: 

ASSEMBLY  PROCESS  MODIFIED  TO  REQUIRE 
PREFOLDING  OF  ALL  INTERBOARD 
FLEXTAPES  PRIOR  TO  INSTALLATION. 

THIS  PROVIDES  STRESS  RELIEF  TO  REDUCE 
_  FORCES  ON  SOLDER  JOINTS. 


b.  FI  00  Field  Data 

Field  data  for  the  F100  components  was  taken  from  the  P&W/GPD  Critical  Components 
Reliability  Assessment  (CCRA)  system.  Shown  in  Table  4  is  a  sample  of  the  data  available  from 
this  system 

This  data  system  currently  contains  over  7000  removal  records  for  the  EEC,  2300  for  the 
MFP,  1000  for  the  CIVV  Actuator,  and  1900  for  the  TT2.5  sensor. 

The  CCRA  system  uses  data  from  multiple  sources  as  shown  in  Figure  5.  The  percentages 
shown  indicate  how  much  each  data  source  contributes  to  the  resulting  CCRA  data  base. 

c.  Advanced  Controls  Development  Data 

Data  on  the  CERT  for  the  F100  DEEC  and  JT9D  EEC  was  taken  from  final  reports  by  the 
component  manufacturer.  These  reports  contain  detailed  information  on  each  failure,  including 
information  on  corrective  action  implementation.  The  testing  was  conducted  for  the  purpose  of 
growing  reliability  and  therefore  forms  an  important  data  base  for  model  evaluation.  Table  5 
shows  an  example  of  the  data  summary  available  from  this  testing. 

Data  from  the  EEC  flight  testing  on  JT8D  engines  in  Boeing  727  aircraft  was  obtained  from 
P&W  Commercial  Engineering.  Complete  information  on  each  malfunction,  analysis,  and 
corrective  action  was  included. 

d.  Operating  Hours 

Operating  time  for  the  F100  components  was  taken  from  F100  development  history,  flight 
test  documentation,  and  USAF  operational  flight  hours  records.  Where  only  flight  hours  were 
available,  they  were  converted  to  operating  hours  using  well-documented  historical  conversion 
factors.  All  operating  time  for  F100  data  was  grouped,  i.e.,  the  fleet  time  was  reported  in  monthly 
increments.  Operating  time  from  the  advance  controls  testing  was  taken  from  detailed  test 
reports  and  is  continuous  data,  i.e.,  fleet  time  at  each  failure. 


Status/Component  Summary  66*1  Information 
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Figure  5.  CCRA  Data  Sources 


TABLE  5.  CERT  SUMMARY  DATA  EXAMPLE 


Item 

No 

MR 

No. 

Unit 

No 

Fault 

Date 

UUT  Time  to 
Malfunction 

AU  UUT 
Time  to 
Malfunction 

Malfunction 

Cause 

Failure 

Classification 

Corrective 

Action 

1 

E001 

6 

01/07/81 

2.0 

4.0 

Socket  Damage 

EMW 

Add  HS9575  Req’ts 

2 

E002 

3 

01/07/81 

10.4 

20.8 

TPT2.Intenn.Unk. 

Other 

Monitor  and  TBD 

3 

E003 

2 

01/13/81 

21.9 

295.0 

Socket  Damage 

EMW 

Add  HS9576  Req’ts 

4 

E004 

5 

01/13/81 

22.5 

295.0 

Socket  Damage 

EMW 

Add  HS9575  Req’ta 

5 

E005 

4 

02/17/81 

628.0 

2459.0 

Socket  Damage 

EMW 

Add  HS9575  Req’ts 

6 

E006 

3 

03/04/81 

1070.0 

3100.0 

Socket  Damage 

EMW 

Add  HS9575  Req’ts 

7 

E008 

6 

05/12/Cl 

1351.1 

7274.0 

Incorrect  tool  use 

EMW 

Added  info  +  QC  test 

8 

E009 

2 

06/08/81 

1735.7 

8996.3 

Socket/lead  contact 

EMD 

Process  change 

9 

E012 

4 

08/18/81 

3466.0 

15,522.1 

Socket/lead  contact 

EMD 

Process  change 

10 

E013 

3 

09/29/81 

4651.0 

19,988.7 

Lead  abrasion 

EMD 

In  process,  TBD 

3.  Review  and  Classification  of  Failure  Data 

All  reported  malfunctions  in  the  data  base  were  reviewed  and  classified  using  the  following 
criteria: 

•  Confirmed  —  The  malfunction  was  confirmed  to  be  a  component  malfunc¬ 
tion  requiring  repair  and/or  corrective  action. 

•  Induced  —  The  malfunction  was  induced  by  the  engine,  other  controls 
components,  test  stand  operation,  mishandling,  or  misassembly. 

•  Unconfirmed  —  The  malfunction  did  not  result  in  an  identified  component 
problem  and  required  no  component  repair. 


•  Logic  —  The  malfunction  was  due  to  a  logic  problem  which  required  changes 
to  the  program  memory  (PROM)  to  correct  schedules,  fault  accommodation, 
etc. 

For  the  purposes  of  reliability-growth  model  assessment,  only  confirmed  failures  were  used. 
Failures  other  than  confirmed  are  not  necessarily  a  function  of  the  true  component  reliability 
and  were  therefore  eliminated  from  the  analysis.  An  example  of  the  resulting  data  base  used  for 
all  further  analysis  L  shown  in  Table  6. 

TABLE  6.  DATA  LISTING  FOR 

GROWTH  MODEL  INPUT; 

EXAMPLE  -  EEC 

Serial  No.  Date  Hours  Status  Location 

AESC  0113  741201  C  mfirmed  Unknown 

AESC  0206  760520  00168  Confirmed  Luke 

AESC  0503  770721  00308  Confirmed  Langley 

AESC  1393  801220  00527  Confirmed  McDUl 

AESC  0368  820225  00463  Confirmed  McDiU 

This  data  base  includes  the  following  number  of  failures  for  each  listed  component  of  the 
F100  control  system  covering  development,  production,  flight  test,  and  field  data  from  1969 
through  1982: 

Component  No.  of  Failures 

EEC  2202 

MFP  415 

CIVV  CTL  884 

TT2.5  Sensor  293 

Data  for  the  F100  EEC  was  further  analyzed  for  use  in  the  Endles3-Burn-In  (EBI)  model 
evaluation.  This  model  is  limited  to  failures  of  solid-state  electronics.  Those  failures,  due  to  solid- 
state  electronics,  were  identified  and  extracted  from  the  F100  EEC  data.  In  order  to  expand  the 
data  base  for  EBI  assessing,  field  data  for  the  F100  Engine  History  Recorder  (EHR)  was  also 
reviewed  and  used  for  evaluation. 

4.  Subcontractor  Participation 

As  a  part  of  this  program,  both  Hamilton  Standard,  FI  00  EEC  supplier,  and  Chandler 
Evans,  F100  MFPS  supplier,  participated  in  reviewing  the  failure  data  and  reliability-growth 
models  being  evaluated.  Each  supplier  was  provided  with  all  failure  data  on  their  component. 
The  suppliers  then  reviewed  that  data  and  provided  revisions  and  corrections  where  required.  All 
comments  were  incorporated  into  the  data  bases  for  the  EEC  and  MFPS. 

In  addition  to  data  review,  these  suppliers  reviewed  the  growth  models  and  work-to-date  on 
the  program. 

5.  Initial  Evaluation  of  Data 

The  initial  evaluation  of  failure  data  involved  plotting  the  data  as  MTBF  versus  time  and 
reviewing  the  resulting  trends.  Questions  which  were  being  investigated  include:  (1)  do  hardware 
changes  explain  trends,  (2)  are  other  influences  such  as  operational  usage  or  maintenance 
practices  influencing  the  data,  and  (3)  where  do  different  program  phases  start  and  stop.  Figure  6 
shows  the  results  of  this  evaluation  for  the  F100  EEC. 


Major  design  changes  to  the  EEC,  changes  in  environment  and  changes  in  maintenance 
personnel,  are  indicated  on  this  plot.  Correlation  between  design  changes  and  reliability  growth 
were  evident  in  mid- 1975  when  shock  mounts  were  installed  on  the  EEC.  In  mid-1979,  the 
cooling  environment  was  improved  with  no  immediate  shift  in  the  growth  curve.  This  is  thought 
to  be  due  to  the  amount  of  inertia  (large  number  of  units,  hours,  and  failures)  in  the  data  system 
and  the  slow  retrofit  program  used  to  install  the  change  (still  incomplete  in  mid-1983). 

A  second  evaluation  was  made  to  investigate  the  check  mark  theory.  This  theory  states  that 
following  introduction  to  service,  the  reliability  of  a  component  drops  off  sharply  and  then 
recovers  to  near  the  original  growth  trend.  To  evaluate  this  phenomenon,  failure  rate  versus  time 
plots  for  the  four  F100  components  were  made  as  shown  in  Figures  7  and  8.  Introduction  to 
service  for  these  components  occurred  in  late  1974.  A  check  mark  on  these  plots  would  be  seen  as 
an  upward  spike  near  this  point  in  time.  As  can  be  seen  from  these  plots,  no  real  evidence  of  a 
check  mark  exists  in  the  data. 
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Figure  8.  GAPCEEC  Cumulative  Failure  Rate  Analysis 
and  TT2.5  Sensor  (Bottom  Photo) 


—  C1VV  Actuator  (Top  Photo) 
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B.  EVALUATION  OF  THE  MODELS 


The  primary  objective  of  this  part  of  the  study  was  to  evaluate  the  five  models  (AMSAA, 
Modified  Duane,  Cox-Lewis,  EBI,  and  Box-Jenkins  Time  Series)  in  a  comparative  manner, 
relative  to  the  criteria  established  in  the  research  and  selection  phase  (Appendix  B)  and 
determine  which  models  are  the  most  appropriate  fc*  use  in  modeling  the  reliability  growth  of 
electronic  engine  controls. 

1.  Time  Series  Analysis  Study 

a.  Objac  tfves /Background 

The  objective  of  this  part  of  the  study  was  to  assess  the  usefulness  of  ARIMA 
(Autoregressive  Integrated  Moving  Average)  models  in  forecasting  controls  reliability.  The  first 
five  criteria  involve  the  models  ability  to  accurately  forecast  reliability  at  some  future  point  in 
time.  For  the  deterministic  (EBI)  model  and  the  NHPP  (AMSAA,  Modified  Duane  and  Cox- 
Lewis)  models,  the  form  of  the  model  and  the  reliability  parameter  (MTBF,  failure  rate  etc.)  were 
prespecified.  For  Box-Jenkins  Time  Series  modeling  a  separate  study  was  required  to  determine 
the  form  of  the  model  and  the  appropriate  reliability  parameter  to  be  modeled. 

b.  The  Data  Used  In  this  Analysis 

The  data  consisted  of  incident  counts  and  total  operating  time  by  month  for  each  of  the 
four  main  components  (EEC,  MFP,  CIW  Actuator,  and  TT2.5  sensor)  operating  in  develop¬ 
ment,  production,  flight  and  field  environment.  In  addition  the  digital  electronic  engine  control 
(DEEC)  data  f~nm  combined  environment  reliability  test  (CERT)  and  the  engine  history 
recorder  (EHR)  field  data  were  used,  as  shown  in  Table  7. 


TABLE  7.  DATA  SETS  AVAILABLE  FOR  ANALYSIS 


Development 

Production 

Field 

Combined 

EEC 

X 

X 

X 

X 

EEC  (solid  state  only) 

X 

X 

X 

X 

Main  Fuel  Pump 

X 

X 

X 

X 

C1VV  Actuator 

X 

X 

X 

X 

TT2.5  Sensor 

X 

X 

X 

X 

Engine  History  Recorder 
DEEC 

X 

X 

Also,  EEC  Flight  Test  Data 

c.  Methodology 

For  determining  the  best  time  series  model  and  the  most  appropriate  reliability  parameter 
to  be  used  in  time  series  modeling,  the  following  variables  were  considered  as  possible  candidates 
for  use  as  a  measure  of  reliability: 

1.  Cumulative  failure  rate  (Xc)  =  cumulative  incidents/cumulative  operating 
time. 

2.  Cumulative  mean  time  between  failures  (MTBFt.)  =  1/A.,.. 

3.  Instantaneous  failure  rate  (A.;)  =  incidents  for  the  month/operating  time  for 
the  month. 


4.  Instantaneous  mean  time  between  failure  (MTBFj)  =  1/X-,- 

5.  Cumulative  failures. 

6.  Instantaneous  failures  by  month. 

While  it  was  unlikely  that  the  last  two  variables  would  be  serious  candidates, 
they  were  included  to  verify  the  need  for  a  time  factor. 

The  four  ARIMA  models  reviewed  in  this  study  are  as  follows: 

1.  ARIMA  (0,1,0)  A  first  difference  model. 

2.  ARIMA  (0,1,1)  A  first  difference  moving  average  model. 

3.  ARIMA  (1,1,0)  A  first  difference  autoregressive  model. 

4.  ARIMA  (1,1,1)  A  first  difference  autoregressive  moving  average  model. 

These  terms  are  discussed  further  in  Appendix  E. 


For  each  variable,  model,  and  data  set  the  model  is  fit  to  the  first  18  months  of  data 
(18  months  after  the  first  failure),  a  forecast  of  the  reliability  is  made  for  the  next  12  months,  and 
summary  statistics  are  computed.  Next  the  model  is  fit  to  18  plus  6  months  worth  of  data  and  the 
process  is  repeated.  This  methodology  is  graphically  illustrated  in  Figure  9.  The  summary 
statistics  used  for  evaluating  the  forecasts  are  as  follows: 

•  Root  Mean  Square  Error  (RMSE) 

D,(OT  I-  E  (Predicted  Reliability  -  Observed  Reliability)2  1  \n 
KMoti  - 12~(months)  J 

•  Largest  Absolute  Error  =  Max  (Predicted  Reliability  —  Observed  Reliability) 

•  Mean  Delta  Percent  Error 


£  (Predicted  Reliability  -  Observed  Reliability 
12  (months) 


Mean  Absolute  Error 


£  Absolute  Value  (Predicted  Reliability  -  Observed  Reliability^ 

12  (months) 


A  preliminary  analysis  indicated  that  the  summary  statistics  were  highly  correlated  (95%), 
therefore  only  the  RMSE  was  analyzed  in  detail.  The  RMSE  was  selected  because  of  its 
popularity  in  the  literature  and  because  it  is  the  most  stable. 


Multivariate  cluster  analysis  techniques  were  initially  used  to  analyze  these  RMSE 
statistics,  and  identify  the  best  model  variable  combination.  The  extreme  overlap  of  clusters 
rendered  a  choice  of  the  best  group  impossible,  thus  indicating  the  need  for  a  finer  analytical 
approach. 
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9.  Methodology  for  Computation  of  Errors 
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The  most  detailed  and  adaptable  statistical  technique  available  for  analysis  of  experimental 
data  is  the  Analysis  of  Variance.  Three  important  assumptions  in  an  Analysis  of  Variance  Study 
are  that  the  errors  are  from  a  normal  distribution,  have  a  common  variance,  and  are  independent 
of  each  ot'ier.  Normality  and  homogeneity  of  variance  were  approximated  after  performing  a  log 
transformation  of  the  data.  The  24  variable/model  (6X4)  combinations  were  made  comparable  by 
standardizing  each  cell  to  a  population  mean  goal  of  zero  error  (i.e.,  X  (observation  *  0 
goal)/SDcell).  This  converted  all  data  to  dimensionless  standard  deviations  from  a  goal  of  zero 
and  made  it  amenable  to  various  combinational  comparisons. 

In  some  cases  (variable/model/data  base/size)  parameter  estimates  failed  to  converge  or 
converged  to  obviously  wrong  points  and  were  classified  as  solution  failures.  The  number  of 
solution  failures  for  each  variable/model  combination  was  used  to  compute  a  weight  variable  for 
each  cell  of  the  form. 

Wt  =  1.0  -  No.  Solution  Failures/(No.  Solution  Failures  +  No.  Solutions). 

Because  of  the  missing  data  (i.e.,  solution  failures),  the  experiment  was  classified  as  an 
unbalanced  design  or  an  analysis  of  variance  (ANOVA)  model  of  less  than  full  rank.  Ultimately, 
an  unbalanced,  weighted,  two-way  (variable/model)  ANOVA  was  solved  to  obtain  the  desired 
analytical  comparisons. 

d.  Model  Performance 

Table  8  shows  the  general  ANOVA  table  for  this  experiment.  The  ANOVA  model,  which  is 
a  linear  model  describing  all  the  variable/model  combinations,  accounts  for  most  of  the  variation 
present  in  the  RMSE  points  (R-Square  =  0.942).  Examination  of  the  residuals  of  the  ANOVA 
model  fit  indicated  that  the  basic  assumptions  (normality,  homogeneity  of  variance,  and 
independence)  were  reasonable. 


TABLE  8.  OVERALL  ANOVA 


Source 

DF 

Sum  of  Squcrcs 

Mean  Square 

F  Value 

Pr  F 

ANOVA  Model 

23 

132,160.068 

5746.099 

5590.02 

0.0001 

Error 

7871 

8090.747 

1.028 

Corrected  Total 

7894 

140,250.810 

R-square: 

0.942 

Initial  analysis  of  the  ANOVA  model  indicate  that  there  is  a  significant  difference  between 
some  or  all  of  the  six  variables  tested  (P  =  0.0001),  there  is  a  significant  difference  between  some 
of  the  variable/model  combinations  (P  =  0.0238),  and  no  significant  difference  between  the 
ARIMA  models  (P  =  0.1557)  as  summarized  in  Table  9.  (Note:  P  should  be  less  than  0.1  to  detect 
a  90%  significance  level.) 


TABLE  9.  DETAILED  ANOVA 


Source 

DF 

Type  III  SS* 

F  Value 

Pr  F 

Test  Variable 

5 

132,126.228 

25,707.53 

0.0001 

ARIMA  Model 

3 

5.349 

1.73 

0.1557 

Variable  x  Model  Interaction 

15 

28.457 

1.85 

0.0238 

*  Type  III  oum  of  Squares  is 
with  missing  data. 

specifically  designed  to  handle  ANOVA  analyses 

Using  Least  Squares  (LS)  Means  (a  method  whereby  missing  data  —  solution  failures  — 
are  compensated  for  by  projecting  what  a  cell  mean  would  be  if  there  were  no  missing  data)  and 
testing  the  null  hypothesis  of  no  differences  between  cell  means,  the  variable  Cumulative 
Failures/Cumulative  Engine  Flight  hours  (cumfail/CEFH  or  was  found  to  generate  the  least 
amount  of  error  over  all  data  bases,  ARIMA  models  and  data  base  sizes.  The  LS  means  are 
shown  in  Table  10.  Each  LS  mean  is  significantly  different  from  each  of  the  others.  Recall  that 
these  values  are  in  terms  of  the  number  of  standard  deviations  away  from  a  goal  of  zero. 

TABLE  10.  LEAST  SQUARE  MEANS  FOR 
VARIABLES 


Test  Variable 

Least  Squares 
Mean 

Cumfail/CEFH 

0.000002108 

Failures 

0.000065415 

Failures/EFH 

1.47617 

Cumfail 

96.3180 

l/(Cumfail/CEFH) 

8253.4604 

l/(Failures/EFH) 

242,541.4670 

The  Xc  variable  satisfies  one  of  the  assumptions  of  ARIMA  modeling:  That  time  intervals 
are  equal.  By  using  rates  as  input  to  ARIMA  modeling,  this  interval  equivalence  is  obtained. 

In  order  to  discern  which  XC/ARIMA  model  combinations  were  generating  significant 
interactions,  orthogonal  comparisons  or  contrasts  were  computed  and  listed  in  Table  11.  Using 
the  ANOVA  contrast  results  for  the  summary  statistics  Largest  Absolute  Error  and  Mean 
Absolute  Error  in  conjunction  with  those  for  RMSE,  the  clearest  path  through  all  the 
overlapping  significant  and  nonsignificant  differences  appears  to  be  the  following: 

•  Xc  ARIMA  (0,1,0)  equivalent  to  Xc  ARIMA  (0,1,1) 

•  X,c  ARIMA  (1,1,0)  equivalent  to  Xc  ARIMA  (1,1,1) 

•  Xc  ARIMA  (0,1,0)  and  A.c  ARIMA  (0,1,1)  significantly  different  from 
Xc  ARIMA  (1,1,0)  and  Xc  ARIMA  (1,1,1) 


TABLE  11.  CONTRASTS  OF  ARIMA 
MODELS 


Contrast  (for  \) 

l)F 

.S’S 

F  Value 

Pr  F 

0,1,0  vs  others 

1 

3.003 

2.92 

0.0874 

0,1,0  vs  0,1,1 

1 

0.194 

0.19 

0.6641 

0,1,0  vs  1,1,0 

1 

4.051 

3.94 

0.0472 

0,1,0  vs  1,1,1 

1 

3.201 

3.11 

0.0777 

0,1,1  vs  1,1,0 

1 

2.473 

2.41 

o.ir^ 

0,1,1  vs  1,1,1 

1 

1.822 

1.77 

0.1831 

0,1,0y0,1,1  vs  1,1,0 

1 

4.283 

4.17 

0.0413 

0,1,0y0,1,1  vs  1,1,0 
1,1,1 

1 

5.651 

5.50 

0.0191 

As  to  which  cumfail/CEFH:ARIMA  model  combination  is  best  regarding  generation  of  the 
least  amount  of  error  in  a  12-month  forecast,  Table  12  shows  that  ARIMA  (1,1,0)  and  ARIMA 
(1,1,1)  are  mildly  better  than  ARIMA  (0,1,0)  and  ARIMA  (0,1,1).  On  the  other  hand,  comparison 
of  the  LS  means  shown  in  Table  13  indicates  that  overlap  is  such  that  there  may  not  be  a  best 
answer. 
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TABLE  12.  LEAST  SQUARE  MEANS  FOR  Xc 
ARIMA  MODELS 


Xc  ARIMA  Model 

Lease  Square  Mean 

0,1,0 

0.000002543 

0,1,1 

0.000002353 

1.1, c 

0.000001781 

1.1.1 

0.000001852 

TABLE  13.  COMPARISON  OF  LS  MEANS 
FOR  Xc  ARIMA  MODELS 


l‘r  /T/Ho:LS  Mean  (l)  =  LS  Mean  (J) 


t/J 

1 

2 

3 

4 

1 

• 

0.6641 

0.0472 

0.0777 

2 

0.6641 

• 

0.1209 

0,1831 

3 

0.0472 

0.1209 

• 

0.8281 

4 

0.0777 

0.1831 

0.8281 

• 

1  = 

0,1.0:  2  = 

0,1.1:  3  = 

1,1,0;  4- 

1.1.1 

e.  Summary 

Since  all  solution  failures  were  generated  by  ARIMA  (1,1,0)  and  ARIMA  (1,1,1)  models,  the 
ARIMA  (0,1,0)  (also  known  as  a  random  walk)  and  the  ARIMA  (0,1,1)  (first  difference  moving 
average)  were  considered  to  be  the  best  models.  According  to  Box  and  Jenkins"'  any  ARI**A 
(0,1,1)  can  be  viewed  as  an  ARIMA  (0,1,0)  buried  in  white  noise.  Therefore,  the  ARIMA  (0,i,l) 
was  selected  for  use  in  the  rest  of  the  GAPCEEC  study.  In  symbolic  terms,  this  model  is: 

Y,  -  Yt_  i  =  0O  +  at-  0,at.  p 

where  Y  is  cumfail/CEFH, 

0O  is  a  drift  constant, 

0,  is  a  moving  average  parameter, 

and  the  a’s  are  random  shocks  to  the  process. 

The  use  of  the  ARIMA  (0,1,1)  is  also  supported  by  research  completed  by  Dr.  Singpurwalla(2). 

2.  Forecasting  Future  Reliability 

The  first  five  criteria  as  referenced  in  Appendix  B,  part  2,  are  concerned  with  the  models’ 
ability  to  forecast  reliability  over  some  future  time  interval. 

a.  Near-Term  Forecast 

The  .  t  criteria  considered  was  the  models’  ability  to  accurately  forecast  the  average 
reliability  over  the  next  12  month  time  period.  The  following  methodology  was  used  to  assess  the 
models  precision  in  making  near-term  (12  month)  forecast: 

111  Box,  G.E.P  ,  and  G.  M.  Jenkins,  Time  Series  Analysis: Forecasting  and  Control,  Holden-Day,  San  Francisco,  1976. 

121  Singpurwalla,  N.  D.,  “Time  Series  Analysis  and  Forecasting  of  Failure  Rate  Process  in  Reliability  and  Fault  Tree 
Analysis:  Theoretical  and  Applied  Aspects  of  System  Reliability  and  Safely  Assessment,"  SIAM,  1975,  pp.  483-507. 
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Start  with  the  first  18  months  of  data  and 

( 1 )  Estimate  the  model  parameters, 

(2)  Calculate  the  predicted  reliability  for  each  of  the  next  12  months, 

(3)  Compute  the  model  precision  (RMSE,  Largest  Absolute  Error,  Mean  Delta 
Percent  Error,  and  Mean  Absolute  Error)  as  in  subhead  c. 

(4)  Increase  the  data  base  by  6  months  (18  +  6)  and  repeat  steps  1  through  4 
above. 

Each  of  the  five  models  were  evaluated  against  each  of  the  data  sets  already  shown  in 
Table  7.  The  average  precision  was  computed  for  each  model  as  the  data  base  is  as  shown 
graphically  in  Figure  10.  The  mean  and  ±  2  standard  deviations  show  the  expected  variatir  n 
from  data  set  to  data  set  of  the  precision  for  each  model  graphically  shown  in  Figures  11  through 
15.  Based  on  Figure  10,  the  AMSAA/Duane  model,  the  EBI  model  and  the  ARIMA  model  all 
predict  average  reliability  within  ±20%.  To  substantiate  this  result  statistically,  each  of  the  five 
models’  precision  (average  mean  %  Error)  was  compared  to  identify  differences.  Table  14  shows 
the  calculated  v. '  jes  of  the  Z  statistic  (standard  normal  random  variable)  to  be  compared  to  the 
Normal  Table  (3„).  At  the  90%  significance  level,  the  tabled  value  (Zu)is  2.8.  Therefore,  values  in 
Table  14  greater  than  2.8  indicate  the  models  are  different. 


Figure  10.  Forecast  Performance  Over  All  Data  Sets  —  Ali  Models 
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Figure  15.  Forecast  Performance  Over  All  Data  Sets  —  Cox-Lewis  Model 

TABLE  14.  NEAR-TERM  FORECASTING 
COMPARISON 


FD  264299 


AMSAA/ 

Duane 

EBI 

Modified 

Duane 

Cox- 

Lewis 

ARIMA 

4.3* 

6.4* 

13.7* 

12.1* 

AMSAA/Duane 

2.7 

7.1* 

8.6* 

EBI 

4.3* 

3.1* 

Modified  Duane 

1.4 

*  Indicates  that  models  differ  at  the  95%  significance  level. 

These  results  can  be  illustrated  with  a  Venn  diagram 
as  follows: 

OGD  O 


ARIMA  Duane  EBI 


Cox -Lewis 

Modified  Duane 


10-20% 

...  - 

Magnitude  of 
Error 


b.  Comparison  of  Methods  of  Parameter  Estimation 

The  Modified  Duane  model  was  expected  to  forecast  with  as  good  or  better  precision  than 
the  AMSAA/Duane  model.  This  was  expected  because  the  Modified  Duane  model  would  be 
mathematically  equivalent  to  the  AMSAA/Duane  model  if  the  limiting  parameters  were  zero. 

•  Modified  Duane  model 

p(t)  =  xptp-1  -+-  e 

•  AMSAA 

p(t)  -  A-PT?-' 

Since  the  Modified  Duane  model  did  not  forecast  with  the  same  precision  (or  better)  than 
the  AMSAA/Duane  model,  and  since  the  only  obvious  difference  was  in  the  method  of  parameter 
estimation,  a  comparison  was  made  of  the  AMSAA  models  forecasting  precision  using 
(1)  regression  estimators  and  (2)  maximum  likelihood  (ML)  estimators.  The  ML  estimators  are 
the  estimates  which  are  most  likely  (from  a  probability  point  of  view)  to  have  generated  the 
observed  data.  ML  estimators  usually  fit  the  most  recent  data  better  than  regression  estimates. 
The  ML  estimators  were  found  to  be  much  better  for  forecasting  future  reliability  (Figures  16 
and  17).  The  ML  estimates  for  the  AMSAA/Duane  model,  when  working  with  grouped  data,  are 
presented  in  MIL-HDBK-189,  Appendix  D.  It  would  be  necessary  to  solve  a  system  of  nonlinear 
equations  to  find  ML  estimates  for  the  modified  Duane  and  Cox-Lewis  models  (Appendix  C). 
Because  of  the  difficulty  in  obtaining  ML  estimates  for  the  Cox-Lewis  and  Modified  Duane 
models,  regression  estimates  were  used.  This  is  one  possible  reason  for  their  poor  performance. 
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Figure  16.  Comparison  of  Methods  of  Parameter  Estimation 
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Figure  17.  Comparison  of  Methods  of  Parameter  Estimation 

c.  Long-Term  Forecast 


Criteria  2  through  4  (Appendix  B,  Part  2)  are  concerned  with  the  models  ability  to  forecast 
the  reliability  of  a  specific  time  point  (entry  into  the  field)  and  over  a  long  period  of  time  (from 
development  to  maturity).  To  address  these  questions,  a  study  was  undertaken  with  the  following 
objectives: 


(1)  To  determine  the  precision  with  which  each  model  could  forecast  reliability 
at  entry  into  the  field  and  project  reliability  at  maturity  from  development 
data. 


(2)  To  determine  the  lead  time  required  to  project  reliability  at  entry  into  the 
field  and  project  reliability  at  maturity  within  ±  10%. 


The  methodology  used  for  the  first  objective  (predicting  field  entry  and  maturity  from 
development)  was  as  follows: 


(1)  Break  the  data  into  two  parts,  total  experience  prior  to  entry  into  the  field, 
and  field  experience  shown  in  Figure  18. 


(2)  Estimate  the  model  parameters  using  the  prior  to  entry  into  the  field 
experience. 


(3)  Forecast  points  for  each  6th-month  of  field  experience. 

(4)  Calculate  the  percent  error  for  each  point 


Forecast  -  Actual 
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Figure  18.  Long-Term  Forecast 


Entry  into  the  field  (30,000  hr)  was  the  first  prediction  made  (6  month)  and  maturity 
(1  million  hr)  was  month  68.  Figures  19  through  23  show  the  results  of  this  study  for  each  model 
with  each  component.  The  AMSAA/Duane  model  and  Time  Series  did  the  best  (within  ±  20%) 
at  entry  into  the  field.  This  is  consistent  with  near-term  forecasting  results.  None  of  the  models 
did  an  acceptable  job  of  forecasting  to  maturity  from  development. 
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Figure  19.  Duane  Analysis  Long-Term  Forecast 
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Figure  22.  Modified  Duane  Analysis  Long-Term  Forecast 
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Figure  23.  Cox- Lewis  Analysis  Long-Term  Forecast 


The  methodology  used  for  the  second  objective,  determining  required  lead  time,  was  as 
follows: 


Starting  with  the  first  18  points, 


(1)  Estimate  the  model  parameters  from  the  data. 


(2)  Forecast  the  reliability  at  entry  into  the  Held  (30,000  hr)  and  maturity 
(1  million  hr). 

(3)  Compute  the  percent  error  as  before. 

(4)  Increase  the  sample  size  by  one  point  (move  closer  to  the  entry  point  or 
maturity)  and  repeat  steps  1  through  4  above. 

Figures  24  through  28  show  the  results  of  lead  time  required  to  forecast  entry  into  the  field 
(30,000  hr).  The  ARIMA  model  and  the  AMSAA/Duane  model  were  able  to  forecast  within 
±  10%  2  to  3  months  in  advance.  The  other  models  were  not  successful  at  obtaining  these  goals. 
Figures  29  through  33  show  the  results  of  lead  time  requirements  for  forecasting  maturity.  The 
ARIMA  and  Duane  models  were  able  to  forecast  within  ±  10%  approximately  1  year  prior  to 
maturity. 
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Figure  24.  Duane  Model  Forecast  at  30,000  Hours 

Long-term  forecasting  with  the  EBI  model  was  attempted  only  with  the  EEC  data  set  since 
it  was  the  only  complete  data  set  that  had  solid  state  electronics  failures.  These  failures  were 
sorted  out  and  modeled  separately. 

In  general,  all  of  the  models  were  unreliable  in  forecasting  reliability  beyond  1  to  2  years 
into  the  future.  As  a  result  of  all  of  the  models  poor  performance  in  long-term  forecasting,  a 
method  of  long-term  forecasting  based  on  management  goals,  engineering  and  statistics  is 
recommended  (Appendix  D). 
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Figure  33.  Cox-Lewis  Model  Forecast  at  1  Million  Hours 

3.  Overall  Model  Comparison 

a.  Development  of  Criteria 

Since  the  value  of  each  model  relative  to  specific  criteria  (Appendix  B,  Part  1)  is  somewhat 
subjective,  it  was  necessary  to  rewrite  the  criteria  to  make  them  consistent  with  a  six-point  rating 
scale  frequently  used  in  the  behavioral  sciences*1*  to  evaluate  subjective  data  such  as  interest, 
values,  attitudes,  opinions,  etc. 

Each  criterion  statement  must  be  rewritten  so  that  the  statement  itself  is  relatively  neutral 
and  can  be  answered  completely  by  one  of  the  six  responses  to  make  each  statement  consistent 
with  the  rating  scale  (Appendix  B,  Part  2).  Two  of  the  original  criteria  had  to  be  split  into  two 
statements  each  (criteria  4  and  6)  and  two  were  combined  into  a  single  statement  (criteria  10  and 
11).  Criterion  4,  Physical  Interpretation,  could  be  answered  one  way  for  interpretation  of 
parameters  and  another  way  for  interpretation  of  the  model.  A  similar  problem  was  encountered 
with  criterion  6,  Adaptability.  Criteria  10  and  11  were  combined  into  one  statement  because  the 
results  of  criterion  10  strongly  influences  the  results  of  criterion  11.  This  is  because  most  test  for 
trends  (criterion  11)  involve  influences  (confidence  limits)  on  the  growth  parameter. 

b.  Analysis  and  Methodology  Used  to  Compare  the  Models 

Each  statement  was  assigned  a  value  for  each  model  based  on  experience  and  technical 
information  obtained  in  the  literature  search  (Appendix  A).  Table  15  shows  the  response  to  each 
criterion  for  each  model.  These  data  in  Table  15  were  then  transformed  into  Van  der  Waerden 
scores*2*  and  a  multiple  comparison  analysis  was  done  based  on  Friedman’s  rank  sums*3*.  The 
Van  der  Waerden  scores  were  used  because  of  the  small  sample  size  (less  than  30  statements). 
Friedman’s  rank  sums  is  a  distribution  free  method  of  comparing  all  pairs  of  models 
simultaneously. 


(I1  Nunnally,  J.  C.,  Psychometric  Theory,  McGraw-Hill,  Inc.,  1967,  p.  520. 

,2)  “SAS  Institute  Technical  Report  S-120X,”SAS  Institute,  Inc.,  Cary  N.  C.,  1981. 

Hollander,  M.,  and  D.  A.  Wolfe,  Nonparametric  Statistical  Methods,  John  Wiley  &  Sons,  Inc.,  New  York,  p.  138. 


TABLE  15.  RESPONSE*  OF  MODELS  TO  CRITERIA 


Table  I 

£i  tal<’ ss 

Modified 

Cox- 

Statement 

ARIMA 

Duane 

Burn-In 

Duane 

Lewis 

1 

5 

5 

5 

2 

2 

2 

2 

2 

2 

1 

1 

3 

2 

2 

1 

1 

1 

4 

2 

3 

1 

1 

1 

5 

2 

5 

2 

2 

2 

6 

4 

3 

5 

4 

4 

7 

2 

5 

4 

4 

4 

8 

1 

5 

6 

5 

4 

9 

1 

5 

5 

4 

4 

10 

5 

3 

3 

3 

2 

11 

6 

4 

2 

4 

4 

12 

5 

5 

2 

2 

2 

13 

5 

2 

2 

2 

2 

14 

1 

1 

5 

5 

2 

15 

4 

5 

4 

2 

3 

*  Possible  responses  to  criteria: 

1 .  Completely  disagree 

2.  Mostly  disagree 

3.  Slightly  disagree 

4.  Slightly  agree 

5.  Mostly  agree 

6.  Completely  agree. 

c.  Results 

The  AMSAA/Duane  model  had  the  highest  over  all  score,  but  it  was  not  significantly 
different  statistically  from  the  ARIMA  model.  The  EBI  model  was  similar  to  the  ARIMA  model 
but  not  as  effective  as  the  AMSAA/Duane  model.  The  EBI  model  scored  lower  than  the 
AMSAA/Duane  model  on  statements  involving  sample  size  requirements  (criterion  5)  and 
physical  restrictions  (criterion  11).  The  Modified  Duane  model  and  Cox-Lewis  model  did  not 
perform  as  well  in  the  forecasting  statements  (1  through  5).  This  is  due  in  part  to  the  lack  of 
estimation  techniques  that  are  available  for  those  models  when  the  data  are  grouped  by  month. 
Table  16  shows  the  results  of  the  simultaneous  pairwise  comparison.  Values  inside  the  interval 
(-1.65, 1.65)  indicate  the  models  are  not  different. 

C.  RECOMMENDATIONS  FROM  EVALUATION 

As  a  result  of  the  work  done  in  the  model  evaluation  phase  of  this  contract,  the  AMSAA 
model  was  considered  to  be  the  most  appropriate  model  for  future  consideration  with  the  ARIMA 
model  as  an  alternative  (and  in  some  cases  the  EBI).  Refer  to  Appendixes  D  and  E  for  more 
information  relative  to  selection  and  implementation  of  these  models. 
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TABLE  16.  RESULTS  OF  MOD¬ 
EL  COMPARISON 


AMSAA/ 

Duane 

EBI 

Modified 

Duane 

Cox- 

Lewis 

ARIMA 

-1.55 

0.202 

1.46 

2.02 

AMSAA/Duane 

1.75 

3.02 

3.58 

EBI 

1.26 

1.82 

Modified  Duane 

0.56 

These  results  are  illustrated  with  a  Venn  diagram 


AMSAA 

cz 


Modified  Duane 

^SE&rj 


Cox-Lewis 


If  two  models  overlap,  they  are  not  different  at  the  90% 
significance  level. _ _ 
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SECTION  IV 


PROGRAM  SYNTHESIS 

A.  INTRODUCTION 

The  program  synthesis  phase  of  this  study  consisted  of  two  primary  objectives:  (1)  To 
research  confidence  limits  and  goodness-of-fit  tests  available  for  the  AMSAA  and  Time  Series 
models;  and  (2)  To  develop  a  procedure  to  be  used  as  a  guide  for  applying  the  AMSAA  and  Time 
Series  model  to  a  development  program. 

B.  CONFIDENCE  LIMITS  AND  GOODNESS-OF-FIT  TEST 

A  thorough  review  of  the  literature  published  within  the  past  10  years  on  confidence  limits 
and  goodness-of-fit  tests  for  the  Weibull  process  (AMSAA  model)  and  Box-Jenkins  Time  Series 
modeling  was  completed  (see  Reference  Subsection  4).  The  confidence  limits  and  goodness-of-fit 
tests  recommended  in  the  GAPCEEC  study  are  consistently  used  and  referenced  in  the 
literature.  The  following  are  recommended: 

1.  For  the  AMSAA  Model,  With  Time-Truncated  Data 

The  exact  unit  and  fleet  time  is  known  at  each  failure  and  some  test  time  has  been  run  since 
the  last  failure.  The  following  are  recommended: 

•  Goodness-of-fit  test.  The  Cramer  von-Mises  test  is  recommended 
(Appendix  D).  Reference  7,  pp.  70  and  135. 

•  Confidence  limits  for  the  mo-del  parameters,  P  and  X.  Reference  7,  p.  68  and 
Reference  5,  respectively. 

•  Test  for  growth.  Use  the  confidence  limits  on  the  P  as  illustrated  in 
Reference  7,  p.  68. 

•  Confidence  limits  on  reliability.  Confidence  limits  are  available  for  the 
instantaneous  MTBF.  Reference  2. 

2.  For  the  AMSAA  Model,  With  Grouped  Data 

Failures  and  fleet  time  tracked  in  discrete  intervals,  for  example,  by  month.  The  following 
are  recommended: 

•  For  goodness-of-fit  and  test  for  growth,  use  a  Chi-Square  test.  Reference  7, 
pp.  68  and  140,  and  p.  69,  respectively. 

•  There  are  no  easy  to  apply  techniques  for  confidence  limits  on  the 
parameters  or  on  reliability  available  at  this  time. 
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3.  For  Box-Jenkins  lime  Series  Analysis 


There  is  no  goodness-of-fit  test  for  a  specified  model  (first  difference  moving  average 
model).  There  are  tests  available  to  determine  if  an  unspecified  model  is  needed.  The  following 
test  and  confidence  limits  are  recommended: 

•  Model  identification.  A  Chi-Square  Test  is  available  to  test  the  hypothesis 
that  the  series  is  white  noise  (no  model  needed).  Once  a  model  is  elected,  for 
example,  moving  average  model,  this  test  can  be  used  to  determine  if  the 
residuals  are  white  noise  (model  explains  all  meaningful  variation).  Refer¬ 
ence  13,  p.  297. 

•  Confidence  limits  are  available  for  the  drift  parameter,  the  moving  average 
parameter  and  the  cumulative  failure  rate.  Reference  8,  pp.  194-195, 187-188, 
and  144-146,  respectively. 

C.  PROCEDURE  FOR  USING  THE  AMSAA  MODEL 

In  the  procedure  (Appendix  D)  there  are  five  sections  and  an  Appendix  (Appendix  E).  The 
first  three  sections  deal  with  planning  and  management  of  the  growth  modeling  process,  while 
the  last  two  sections  are  concerned  with  the  analysis  once  the  program  is  underway  and  the  data 
has  been  collected. 

1.  Section  I  —  Preliminary  Analysis  and  Background  Research 

Section  I  gives  a  description  of  the  background  work  necessary  to  plan  a  reliability  growth 
program.  This  section  covers  breaking  the  data  into  test  phases,  reviewing  past  growth  models 
and  growth  rates,  assessing  the  computer  capabilities  and  a  description  and  recommendation  of 
what  kind  of  data  to  collect. 

2.  Section  II  —  Model  Selection 

Section  II  gives  a  detailed  description  of  each  of  the  following  models:  The  AMSAA/Duane 
model,  the  Box-Jenkins  Time  Series  model  and  the  Endless-Burn-In  model.  For  each  of  these 
models,  the  advantages  of  the  model  are  presented  as  well  as  the  form  of  the  model  and  the 
assumptions  of  each  model.  The  AMSAA/Duane  model  is  generally  the  preferred  model. 

3.  Section  III  —  Design  the  Ideal  Growth  Curve 

Section  III  is  concerned  with  illustrating  how  to  draw  the  planned  curve  from  the 
information  obtained  in  the  first  section  of  the  procedure.  This  section  assumes  that  the 
AMSAA/Duane  model  is  being  used.  After  illustrating  how  to  calculate  the  growth  parameter 
and  average  MTBF  for  each  phase,  Section  3  discusses  how  to  evaluate  the  feasibility  of  the  ideal 
curve . 

4.  Section  IV  —  Analysis  of  Data 

Section  IV  illustrates  how  to  perform  engineering  and  statistical  analysis  to  assess  the 
current  reliability  and  to  forecast  reliability  at  points  in  the  future.  This  section  assumes  that  the 
data  has  already  been  gathered.  The  engineering  analysis  subsection  discusses  reviewing  the 
failure  data  from  an  engineering  perspective  to  determine  the  type  of  failure  mode,  and  the  cause 
of  the  failure.  Engineering  assessment  of  the  influence  of  potential  fixes  for  each  failure  mode  is 
discussed.  Statistical  analysis  of  both  grouped  data  the  time-truncated  data  are  reviewed.  This 
analysis  includes:  preliminary  analysis,  parameter  estimation,  confidence  limits,  goodness-of-fit 
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test  and  tests  for  growth,  as  well  as  estimation  of  current  and  future  reliability.  Development  and 
operational  environments  are  considered  in  this  section. 


5.  Section  V  —  Interpretation  of  Results 

This  section  briefly  describes  the  interpretation  of  the  results  of  the  statistical  analysis  in 
light  of  the  original  objectives:  assessment,  prediction  and  control. 

D.  REFERENCES,  BOX-JENKINS  TIME  SERIES 

This  list  of  references  to  the  procedure  is  intended  to  introduce  some  of  the  basic  concepts 
of  ARIMA  modeling  to  the  user  and  to  illustrate  its  applicability  to  reliability-growth  modeling. 
There  are  two  main  sections,  the  first  being  a  discussion  of  the  background,  motivation  for  and 
theory  of  time  series,  and  the  second  being  an  example  of  an  ARIMA  model  applied  to  part  of  the 
EEC  data. 


Articles  Relating  to  Confidence  Limit 
of  the  AMSAA  and  ARIMA  Models 

1.  Belot,  E.  F.,  “A  Computer  Program  for  Estimation  of  Parameters  of  the  Weibull  Intensity 
Function  and  for  the  Cramer  von-Mises  Goodness-of-Fit  Test,”  Technical  Report  No.  279, 
Aberdeen  Proving  Ground,  Md. 

2.  Crow,  L.  H.,  “Confidence  Interval  Procedures  for  the  Weibull  Process  with  Applications  to 
Reliability  Growth,”  Technometrics,  Vol.  24, 1982. 

3.  Crow,  L.  H.,  “Confidence  Interval  Procedures  for  Reliability  Growth  Analysis,”  Technical 
Report  No.  197,  Aberdeen  Proving  Ground,  Md.,  June  1977. 

4.  Crow,  L.  H.,  “Reliability  Analysis  of  Complex  Repairable  Systems,”  Soc.  Industrial  and 
Applied  Mathematics,  Reliability  and  Biometry:  Proceedings  of  Statistical  Analysis  of  Life 
Length,  July  1974. 

5.  Finkelstein,  J.  M.,  “Confidence  Bounds  on  the  Parameters  of  the  Weibull  Process,” 
Technometrics,  Vol.  18,  No.  1, 1976. 

6.  Lee,  L.,  and  S.  K.  Lee,  “Some  Results  on  Inferences  for  the  Weibull  Process,” 
Technometrics,  Vol.  20, 1978. 

7.  MIL-HDBK-189,  “Reliability  Growth  Management,”  1981. 

8.  Box,  G.  E.  P.,  and  G.  M.  Jenkins,  Time  Series  Analysis:  Forecasting  and  Control,  Holden- 
Day,  San  Francisco,  1976,  pp.  91-108, 119-120, 135, 144-146, 176, 187-189. 

9.  Fuller,  W.  A.,  Introduction  to  Statistical  Time  Series  ,  John  Wiley  &  Sons,  New  York,  1976, 
pp.  372-373. 

10.  Anderson,  0.  D.,  Time  Series  Analysts  and  Forecasting  ,  Butterworths,  London,  1976,  pp. 
93-123. 


11.  Cleary,  J.  P.,  and  H.  Levenbach,  “The  Professional  Forecaster,”  Lifetime  Learning 
Publications,  Belmont,  CA,  1982,  pp.  263-282. 
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SECTION  V 

WARRANTY  CONSIDERATIONS 

The  contribution  that  growth  modeling  can  make  to  warranty  of  future  engine  components 
was  investigated  as  part  of  this  program.  Two  types  of  warranties  have  been  used  for  engine  and 
engine  components:  1)  warranty  of  a  level  of  reliability  (mean  time  between  failure  (MTBF))  at  a 
specific  point  in  development;  and  2)  warranty  of  a  specified  period  of  operating  hours  with 
failures  occuring  during  that  period  repaired  at  no  cost  to  the  customer.  These  two  types  of 
warranties,  and  the  growth  models  contribution  to  each,  are  discussed  further  below. 

A.  MTBF  WARRANTY 

Warranty  of  a  specific  level  of  reliability  has  been  applied  to  engine  programs  during  the 
development  process.  This  type  warranty  guarantees  that  a  certain  level  of  reliability,  in  terms  of 
MTBF,  will  be  met  at  the  end  of  a  particular  development  phase.  During  development,  detailed 
data  can  be  collected  and  analyzed. 

The  reliability-growth  modeling  techniques,  which  have  been  developed  in  this  study,  apply 
directly  to  tracking  of  this  development  data,  assessment  of  current  (instantaneous)  MTBF  and 
short-term  projection  of  reliability.  Using  these  techniques,  the  reliability-growth  model  can  be 
used  to  evaluate  the  ability  of  the  current  design  to  meet  the  warranty  value  at  the  required  point 
in  development.  The  current  and  projected  reliability  using  the  model  do  not  provide  the 
statistical  confidence  provided  by  a  demonstration  type  test  and  do  not  replace  the  requirement 
to  demonstrate  contractual  compliance  with  specific  warranty  values. 

B.  FIXED  TIME  WARRANTY 

Warranty  for  a  specific  number  of  operating  hours  following  delivery  has  been  applied  to 
engine  programs  during  operational  field  usage.  This  type  warranty  covers  malfunctions  of  a 
component/engine  for  a  specified  number  of  operating  hours  from  the  date  of  delivery  to  the 
customer.  Provisions  of  this  type  warranty  could  include  repair  at  no  cost,  replacement, 
reimbursement  for  repair  or  others  as  agreed  to.  Knowing  the  past  history  of  j  o  pponent,  the 
current  and  projected  reliability  of  the  component  can  be  calculated  using  the  rehability-growth 
model.  The  current  or  projected  reliability  in  turn  can  be  used  to  calculate  the  expected  returns 
under  a  given  warranty. 

For  this  type  warranty,  an  evaluation  procedure  to  use  the  growth  model  with  existing 
component  data  was  established.  The  purpose  of  this  evaluation  was  to  determine  the  accuracy  of 
model  projections  of  warranty  returns  versus  actual  field  experience. 

Warranty  Evaluation  Procedure 

The  procedure  for  evaluating  the  growth  model’s  ability  to  project  returns  under  a  fixed 
time  type  warranty  situation  consisted  of: 

( 1 )  Assuming  a  warranty  situation 

(2)  Projecting  component  failures  based  upon  prior  data,  using  the  reliability- 
growth  model 

(3)  Determining  the  actual  number  of  failures  from  the  failure  data  base 

(4)  Comparing  the  projected  number  of  failures  to  the  actual  number  of 
failures  during  the  assumed  warranty. 


One  warranty  selected  was  for  the  first  200  hours  of  control  component  field  operation. 
This  is  similar  to  the  warranty  currently  in  place  for  F100  engine  components.  In  addition, 
warranty  periods  of  the  first  300  and  400  hours  were  used  in  the  evaluation. 

Each  assumed  warranty  period  was  applied  to  the  yearly  production  lots  of  components 
from  development  through  current  production.  Knowing  the  warranty  hours  per  unit  and  the 
number  of  units  in  each  production  lot  allows  calculation  of  the  total  hours  of  exposure  for  the 
lot.  For  example,  if  there  were  100  units  in  a  given  production  lot  and  a  200  hour  warranty,  a  total 
of  20,000  hours  of  operation  are  covered  by  the  warranty. 

Component  data  up  to  the  delivery  of  the  first  warranty  units  was  used  in  making  the 
MTBF  projection.  The  data  was  fit  to  the  reliability-growth  model,  model  parameters 
determined,  and  a  calculation  of  instantaneous  MTBF  made  at  delivery  of  the  first  warranty 
units.  This  instantaneous  MTBF  was  used  to  calculate  the  expected  number  of  returns. 

Knowing  the  total  hours  covered  by  warranty  and  the  instantaneous  MTBF,  the  expected 
returns  are  calculated  as  hours  divided  by  MTBF.  These  calculations  were  made  for  each  year’s 
production  lot  considering  all  available  experience  up  to  that  point  in  time.  For  example,  the  first 
year’s  warranty  used  only  development  experience,  the  second  year’s  warranty  used  development, 
data  and  the  first  year’s  production  experience,  and  the  third  year’s  warranty  used  development 
plus  two  years  production,  etc. 

The  component  data  base  was  used  to  determine  the  actual  number  of  failures  from  each 
production  lot  under  the  warranty  ground  rules.  These  actual  failures  were  then  compared  to  the 
projected  failures  to  determine  the  accuracy  and  hence  the  usefulness  of  the  model  with  regard  to 
warranty  evaluation. 

C.  WARRANTY  EVALUATION  RESULTS 

Tables  17,  18,  and  19  show  the  results  of  this  evaluation.  In  these  tables,  PRODUCTION 
LOT  identified  as  development  includes  those  units  used  for  flight  test  evaluation;  1st  Year 
includes  those  units  in  the  first  year’s  production  and  etc.  NO.  UNITS  is  the  number  of  units  in 
the  production  lot.  TOT  HR  is  the  total  hours  covered  by  the  warranty  calculated  as  No.  units 
times  warranty  period.  INST  MTBF  is  the  current  MTBF  from  the  growth  model  using  all 
available  data.  Projected  failures  are  the  number  of  expected  failures  for  the  lot  under  the  given 
warranty  calculated  as  TOT  HR  divided  by  INST  MTBF.  The  OBSERVED  FAILURES  are 
those  that  actually  occurred  with  hours  less  than  the  warranty  limit. 

TABLE  17.  SIMULATED  200  HOUR  WARRANTY 
RESULTS 


Model  Projection 

Projected 

Failures 

Field  Data 

Production 

Lot 

No. 

Units 

Total 

Hr 

Inst 

MTBF 

Observed 

Failures 

Development 

76 

15,200 

1,380 

11 

33 

1st  Year 

173 

34,600 

608 

57 

59 

2nd  Year 

362 

72,400 

576 

126 

74 

Ilrd  Year 

426 

85,200 

813 

105 

72 

•1th  Year 

293 

58,600 

1,014 

58 

43 

Sth  Year 

511 

102,200 

1,268 

81 

58 

6th  Year 

582 

1 16,400 

790 

147 

44 

7th  Year 

796 

159,200 

678 

235 

Incomplete  Data 

8th  Year 

668 

133,600 

816 

164 

Incomplete  Data 

TABLE  18.  SIMULATED  300  HOUR  WARRANTY 
RESULTS 


Model  Projection 

Field  Data 

Production 

No. 

Total 

Inst 

Projected 

Observed 

Lot 

Units 

Hr 

MTBF 

Failures 

Failures 

Development 

76 

22,800 

1,380 

17 

39 

1st  Year 

173 

61,900 

608 

85 

76 

2nd  Year 

362 

108,600 

676 

189 

110 

3rd  Year 

426 

127,800 

813 

157 

120 

4th  Year 

293 

87,900 

1,014 

87 

58 

5th  Year 

511 

153,300 

1,268 

121 

81 

6th  Year 

582 

174,600 

790 

221 

65 

7th  Year 

796 

238,800 

678 

352 

Incomplete  Data 

8th  Year 

668 

200,400 

816 

246 

Incomplete  Data 

TABLE  19. 

SIMULATED  400  HOUR  WARRANTY 

RESULTS 

Model  Projection 

Field  Data 

Production 

No. 

Total 

Inst 

Projected 

Observed 

Lot 

Units 

Hr 

MTBF 

Failures 

Failures 

Development 

76 

30,400 

1,380 

22 

46 

1st  Year 

173 

69,200 

608 

114 

98 

2nd  Year 

362 

144,800 

576 

251 

138 

3rd  Year 

426 

170,400 

813 

210 

171 

4th  Year 

293 

117,200 

1,014 

116 

91 

5th  Year 

511 

204,400 

1,268 

161 

103 

6th  Year 

582 

232,800 

790 

295 

78 

7th  Year 

796 

318,400 

678 

470 

Incomplete  Data 

8th  Year 

668 

267,200 

816 

327 

Incomplete  Data 

From  these  results  it  can  be  seen  that  the  model  does  not  accurately  predict  warranty 
failures.  This  is  largely  due  to  trends  of  the  growth  curve  which  cause  the  projected  MTBF  to  be 
significantly  different  from  the  actual  MTBF.  Figure  34  shows  a  plot  of  the  2nd  years  production 
lot  projection  as  an  example.  The  projection  is  based  upon  all  experience  through  the  1st  year. 
The  value  of  current  MTBF  from  the  model  is  found  to  be  576  hours.  This  value  is  used  for 
warranty  return  calculations.  The  actual  MTBF  as  shown  on  the  plot  increased  rapidly  during 
the  2nd  year  resulting  in  a  much  higher  MTBF  than  originally  predicted.  This  results  in 
significantly  fewer  returns  than  were  projected. 


The  reliability-growth  model  can  be  used  as  a  tool  in  predicting  fixed  time  tvpe  warranty 
returns;  however,  the  accuracy  of  these  projections  is  not  high  and  caution  should  be  exercised. 

It  should  also  be  noted  that  the  model  projects  only  returns  for  repair  as  a  result  of 
confirmed  component  removals.  It  does  not  consider  returns  which  may  have  secondary 
(induced)  failures  or  units  when  no  fault  can  be  found  (unconfirmed). 
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SECTION  VI 

RESULTS  AND  CONCLUSIONS 


A.  RESULTS 

1.  Of  all  the  models  considered  (AMSAA/Duane,  Modified  Duane,  Cox-Lewis,  Endless-Bum- 
In,  and  ARIMA)  the  AMSAA/Duane  model  was  considered  the  best  model  for  general  use.  This 
model  was  the  easiest  to  use,  could  be  displayed  graphically,  performed  better  than  most  of  the 
other  models  in  short-  and  long-term  forecasting  and  is  the  most  popular  model  in  the  industry. 

2.  There  are  two  basic  methods  for  estimating  the  parameters  of  the  AMSAA/Duane  model: 
(1)  The  first  is  to  find,  by  regression  analysis,  the  set  of  parameters  that  minimizes  the  sum  of 
the  squared  errors  (actual  —  predicted);  (2)  The  second  method  is  to  find,  (by  likelihood 
equations),  the  set  of  parameters  that  maximizes  the  probability  of  observing  the  actual  data. 
The  second  method,  maximum  likelihood  estimates,  was  the  best  for  forecasting  future 
reliability. 

It  was  determined  that  continuously  tracked  data  is  superior  to  grouped  data  when  using 
the  AMSAA  model.  The  parameters  can  be  easily  and  more  precisely  estimated,  confidence 
limits,  and  goodness-of-fit  tests  are  available,  and  exploratory  analysis  is  simplified. 

3.  Time  Series  ARIMA  modeling  was  evaluated  in  detail  and  determined  to  be  a  feasible 
alternative  to  the  (NHPP)  models  presently  being  used.  The  cumulative  failure  rate  (cum 
failures/cum  time)  was  the  best  parameter  to  use  to  describe  reliability  and  the  ARIMA  (0, 1, 1) 
(first  difference  moving  average)  was  the  best  model  form  to  be  used. 

4.  A  procedure  has  been  written  which  illustrates,  with  examples,  how  to  apply  the 
AMSAA/Duane  model  and  ARIMA  model  to  electronic  controls.  This  procedure  shows  how  to 
develop  to  a  planned  curve,  collect  the  data,  analyze  the  data,  and  interpret  the  results. 

5  The  Endless-Burn-In  model  was  determined  to  be  an  effective  model  for  solid  state 
electronics  reliability  when  enough  time  is  being  obtained  to  see  the  long-term  burn-in  effect  and 
the  average  age  of  the  system  being  run  is  representative  of  the  individual  systems. 

6.  A  list  of  references  included  as  Appendix  A  on  reliability  growth  has  been  compiled  and 
categorized  in  Table  1  by  model  type  (NHPP,  Time  Series,  Deterministic,  etc.)  and  by  article 
content  (Theory,  Application,  Discussion,  etc.). 

7.  Current  state-of-the-art  technology  for  estimating  reliability  jumps  as  a  result  of  delayed 
fixes  was  incorporated  into  this  study. 

8.  The  AMSAA/Duane  model  can  be  used  as  a  tool  in  evaluating  the  reliability  of  components 
for  warranty  purposes;  however,  the  accuracy  of  predicting  the  absolute  number  of  expected 
returns  under  warranty  is  not  high. 

B.  CONCLUSIONS 

1.  The  AMSAA  model  is  the  recommended  model  for  use  in  modeling  military 
aircraft  electronic  engine  controls. 

2.  Maximum  Likelihood  estimation  is  the  recommended  method  for  calculating 
the  parameters  of  the  AMSAA  model. 


3.  Data  should  be  tracked  continuously  on  an  individual  and  fleet  basis. 


4.  Box-Jenkins  Time  Series  modeling  is  an  effective  tool  for  modeling 
reliability  growth  when  the  AMSAA  model  is  not  appropriate. 

5.  The  Endless-Burn-in  model  is  effective  in  modeling  solid-state  electronics 
reliability  when  large  amounts  of  time  are  expected  and  average  unit  age  is 
representative  of  individual  ages. 

6.  Dr.  Crow's  methods  for  estimating  the  effects  of  delayed  fixes  on  reliability 
should  be  considered  as  an  alternative  to  data  purging  (scaling  failure  models 
that  have  fixes  in  the  system). 

7.  Reliability  growth  modeling  can  be  used  as  a  tool  in  assessing  and  projecting 
reliability  for  warranty  purposes.  However,  the  model  does  not  provide  the 
accuracy  required  to  act  as  a  sole  source  of  information  for  basing  warranty 
agreements. 

8.  While  none  of  the  models  reviewed  were  effective  for  assessing  reliability  in 
the  long  term,  the  AMSAA/Duane  model,  the  Endless-Bum-In  model  and 
the  ARIMA  model  were  all  effective  in  making  short-term  forecasts. 


APPENDIX  A 

LIST  OF  APPLICABLE  DOCUMENTS 

This  appendix  is  a  list  of  books,  articles  and  reports  reviewed  for  the  GAPCEEC  program. 
Most  of  these  documents  directly  address  reliability-growth  modeling;  however,  a  few  documents 
address  other  types  of  statistical  methodology  that  were  used  in  the  program  to  analyze  data  but 
not  as  a  growth  model.  Comments  have  been  included  where  appropriate.  This  literature  has 
been  grouped  by  model  type  and  subject  matter  for  easy  reference  as  shown  in  Table  A-l. 

1.  W.  W.  Abadeer,  “Statistical  Evaluation  of  Learning  Factor  in  Reliability  Studies,”  IEE 
Trans.  Reliability,  Vol  R-29,  Dec  1980,  pp  414-415. 

2.  S.  Abe,  “Field  Data  Analysis  Via  Markov  Renewal  Life  Models,”  Proc.  1975  Annual 
Reliability  and  Maintainability  Symp.,  1975,  pp  562-567. 

3.  H.  Akaike,  “Markovian  Representative  of  Stochastic  Processes  and  Its  Application  to  the 
Analysis  of  Autoregressive  Moving  Average  Processes,”  Annals  of  the  Institute  of  Stat. 
Math.,  Vol  26, 1974,  pp  363-387. 

Important  demonstration  of  the  equivalence  of  ARMA  and  Markovian  pro¬ 
cesses.  Gives  derivation  of  the  information  theoretic  criterion  useful  in  ARMA 
system  model  identification. 

4.  G.  Aimassy,  “Limits  of  Models  in  Reliability  Engineering,”  Proc.  1979  Annual  Reliability 
and  Maintainability  Symp.,  1979,  pp  364-367. 

5.  0.  D.  Anderson,  Time  Series  Analysis:  The  Box-Jenkins  Approach,  Butterworths,  London, 
1976. 

Good  introduction,  with  many  examples,  to  ARIMA  modeling. 

6.  0.  D.  Anderson,  Time  Series  Analysis  and  Forecasting,  London,  Butterworths,  1976, 
pp  93-123. 

7.  Anonymous,  “Reliability  Growth  Management,  Testing  and  Modeling”  Seminar  Proc., 
Institute  of  Environmental  Sciences,  Chesapeake  Chapter,  n(83),  1978. 

8.  H.  E.  Ascher,  “Evaluation  of  Repairable  System  Reliability  Using  Bad-as-Old  Concepts,” 
IEEE  Trans.  Reliability,  Vol  R-17,  Jun  1968,  pp  103-110. 

First  paper  to  provide  clarification  and  appropriate  analysis  using  a  nonhomoge- 
neous  Poisson  process  for  studying  the  reliability  growth  of  repairable  mechani¬ 
cal  systems. 

9.  H.  E.  Ascher,  “Distribution-Free  Estimation  of  System  Reliability,”  Proc.  1980  Annual 
Reliability  and  Maintainability  Symp.,  1980,  pp  374-378. 

Exemplifies  the  commonalities  between  biostatisticai  survival  analysis  and 
reliability  engineering  analysis. 
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10.  H.  E.  Ascher,  “Weibull  Distribution  vs  Weibull  Process,”  Proc.  1981  Annual  Reliability 
and  Maintainability  Symp.,  1981,  pp  426-431. 

Provides  justification  for  avoiding  the  phrase  “Weibull  process.” 

11.  H.  Ascher,  H.  Feingold,  “Bad  as  Old  Analysis  of  System  Failure  Data,”  Annals  of 
Assurance  Sciences,  8th  Reliability  and  Maintainability  Conf.,  1969,  pp  49-62. 

12.  H.  Ascher,  H.  Feingold,  “Is  There  Repair  After  Failure?,”  Proc.  1978  Annual  Reliability 
and  Maintainability  Symp.,  1978,  pp  190-197. 

13.  H.  Ascher,  H.  Feingold,  “The  Aircraft  Air  Conditioner  Data  Revisited,”  Proc.  1979  Annual 
Reliability  and  Maintainability  Symp.,  1979,  pp  153-159. 

14.  N.  T.  J.  Bailey,  The  Elements  of  Stochastic  Processes,  John  Wiley  &  Sons,  New  York,  NY, 
1964. 

15.  L.  J.  Bain,  F.  T.  Wright,  “The  Negative  Binomial  Process  with  Applications  to 
Reliability,”  J.  of  Quality  Technology,  Vol  14,  Apr  1982,  pp  60-66. 

Describes  an  alternative  to  the  Poisson  distribution  for  count  data. 

16.  H.  S.  Balaban,  “Reliability  Growth  Models,”  J.  Environmental  Sciences,  Vol  21,  Jan-Feb 
1978,  pp  11-18. 

A  good  review  of  many  of  the  current  used  reliability  growth  models. 

17.  R.  E.  Barlow,  ".inalysis  of  Retrospective  Failure  Data  Using  Computer  Graphics,”  Proc. 
1978  Annual  Reliability  and  Maintainability  Symp.,  1978,  pp  113-116. 

Promotes  the  use  of  total  time  on  test  analysis. 

18.  R.  E.  Barlow,  F.  Proschan,  Statistical  Theory  of  Reliability  and  Life  Testing,  Holt,  Rinehart 
and  Winston,  New  York,  NY  1975. 

A  significant  contribution  to  the  theory  of  statistical  reliability. 

19.  It.  E.  Barlow,  E.  M.  Scheuer,  “Reliability  Growth  During  A  Development  Testing 
Program,'’  i'echnometrics,  Vol  8,  Feb  1966,  pp  53-60. 

Derives  the  trinomial  model  of  reliability  growth. 

20.  D.  R.  Barr,  “Class  of  General  Reliability  Growth  Models,”  Operauuns  RtdKCirch,  Vol.  18, 
1970,  pp  52-65. 

21.  D.  E.  Beachler,  W.  A.  Chapman,  “Reliability  Proving  for  Commercial  Products,”  Proc. 
1977  Annual  Reliability  and  Maintainability  Symp.,  1977,  pp  89-94. 

Application  of  the  Duane  model. 
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22.  E.  F.  Belot,  "A  Computer  Program  for  Estimation  of  Parameters  of  the  Weibull  Intensity 
Function  and  for  the  Cramer  von  Mises  Goodness  of  Fit  Test,  ”  Tech.  Rept.  279,  US  Army 
Material  Systems  Analysis  Activity,  Aberdeen  Proving  Ground,  MD,  Jul  1979. 

23.  A.  Bezat,  F.  Kreuze,  “The  Endless  Burn-In  Growth  Model  for  Projecting  Stress-Screening 
Requirements,  ”  Internal  Rept.,  Avionics  Division,  Honeywell  Inc.,  Minneapolis,  MI,  1982. 

24.  A.  G.  Bezat,  L.  L.  Montague,  “The  Effect  of  Endless  Burn-In  on  Reliability  Growth 
Projections,”  Proc.  1979  Annual  Reliability  and  Maintainability  Symp.,  1979,  pp  392-397. 

Presents  a  modified  geometric  curve  method  with  supporting  data  for  modeling 
reliability  growth  of  solid-state  electronic  systems. 

25.  A.  Bezat,  V.  Norquist,  L.  Montague,  “Growth  Modeling  Improves  Reliability  Predictions,” 
Proc.  1975  Annual  Reliability  and  Maintainability  Symp.,  1975,  pp  317-322. 

26.  R.  Billinton,  M.  Alam,  “Effect  of  Restricted  Repair  on  System  Reliability  Indices,”  IEEE 
Trans.  Reliability,  Vol  R-27,  Dec  1978  pp  376-380. 

Derives  system  average  failure  rate  and  average  outage  duration  via  Markov 
chains. 

27.  Z.  W.  Birnbaum,  On  the  Mathematics  of  Competing  Risks,  Vital  and  Health  Statistics: 
Series  2,  Data  Evaluation  and  Methods  Research,  'll,  DHEW  Publ.,  Washington,  D.C. 
(PHS)  79-1351, 1979. 

Excellent  reference  on  parametric  and  nonparametric  theory  and  estimation 
techniques  relating  to  survival  analysis. 

28.  H.  S.  Blanks,  “Electronic  Reliability:  a  State  of  the  Art  Survey,”  Microelectronics  and 
Reliability,  Vol  20, 1980,  pp  219-245. 

General  discussion  of  the  physics  of  electronic  component  reliability  including 
reliability  growth  with  extensive  bibliography. 

29.  S.  Blumenthal,  J.  A.  Greenwood,  L.  H.  Herbach,  “The  Transient  Reliability  Behavior  of 
Series  Systems  or  Superimposed  Renewal  Process,”  Technometrics,  Vol  15,  May  1973, 
pp  255-269. 

An  interesting  possible  alternative  to  using  nonhomogeneous  Poisson  processes 
to  evaluate  repairable  system  reliability  growth. 

30.  S.  Blumenthal,  J.  A.  Greenwood,  L.  H.  Herbach,  "A  Comparison  of  the  Bad  as  Old  anu 
Superimposed  Renewal  Models,”  Management  Science,  Vol  23,  Nov  1976,  pt>  280-285. 


31.  T.  J.  Boardman,  M.  C.  Bryson,  “A  Review  of  Some  Smoothing  and  Forecasting 
Techniques,”  J.  Quality  Technology,  Vol  10,  Jan  1978  pp  1*11. 

32.  A.  J.  Bonis,  “Reliability  Growth  Curves  for  One  Shot  Devices,”  Proc.  1977  Annual 
Reliability  and  Maintainability  Symp.,  1977,  pp  181-186. 

Uses  a  modified  exponential  function  to  determine  growth. 

33.  G.  E.  P.  Box,  G.  M.  Jenkins,  Time  Series  Analysis:  Forecasting  and  Control,  Holden-Day, 
San  Francisco,  Ca,  1976. 

The  basic  test  on  ARIMA. 

34.  J.  E.  Bresenham,  “Reliability  Growth  Models,”  Tech.  Rept.  74,  N(95),  Defense  Logistics 
Agency,  Alexandria  VA,  Aug  1964. 

An  interesting  early  examination  of  reliability  growth  models. 

35.  J.  J.  Bussolini,  “The  Benefits  of  a  Totally  Integrated  Reliability  Test  Program,”  AGARD, 
Lecture  Series  47  on  Reliability  and  Avionics  Systems,  7  Rue  Ancelle  92200  Nevilly  Sur 
Seine  France,  1971,  pp  11-1  and  11-21. 

36.  J.  K.  Byers,  D.  H.  Galli,  “Reliability  Growth  Apportionment,”  IEEE  Trans.  Reliability, 
Vol  R-26,  Oct  1977,  pp  242-244. 

Demonstrates  the  use  of  a  mathematical  programming  technique  which  incorpo¬ 
rates  cost  data  to  examine  reliability  growth. 

37.  J.  M.  Clarke,  “No-Growth  Growth  Curves,”  Proc.  1979  Annual  Reliability  and  Maintaina¬ 
bility  Symp.,  1979,  pp  407-410. 

Stresses  using  engineering  analysis  in  conjunction  with  statistical  analysis  to 

ensure  accurate  conclusions. 

38.  J.  M.  Clarke,  W.  P.  Cougan,  “A  Recent  Real  Life  Case  History,”  Proc.  1978  Annual 
Reliability  and  Maintainability  Symp.,  1978,  pp  231-242. 

Duane  model  applied  to  electronics. 

39.  J.  P.  Cleary,  H.  Leavenbach,  The  Professional  Forecaster,  Lifetime  Learning  Publications, 
Belmont,  CA,  1982,  pp  263-282. 

40.  E.  O.  Codier,  “Reliability  Growth  in  Real  Life,”  Proc.  1968  Annual  Symposium  on 
Reliability,  1968,  pp  458-469. 

Promotes  use  of  the  Duane  model. 

4i  W.  J.  Corcoran,  II.  Weingarten,  P.  W.  Zehna,  “Estimating  Reliability  After  Corrective 
Actons,”  Management  Science,  Vol  10,  Jul  1964,  pp  786-795. 

Demonstrates  a  method  for  purging  data  and  estimating  current  reliability  after 

corrective  action  based  on  the  binomial  distribution. 


42.  W.  J.  Corcoran,  R.  R.  Read,  “Study  of  the  Reliability  Growth  Implications  of  Subsystem 
vs  Full  Assembly  Testing,”  United  Technologies  Research  Center,  UTC  2140-FR,  United 
Aircraft  Corporation  Sunnyvale,  Calif,  Nov  1968. 

43.  D.  R.  Cox,  “Regression  Models  and  Life  Tables,”  J.  Royal  Stat.  Soc.,  Series  B,  Vol  34, 
1972,  pp  187-220. 

A  milestone  in  survival  data  analysis  methods. 

44.  D.  R.  Cox,  P.  A.  W.  Lewis,  The  Statistical  Analysis  of  Series  of  Events,  Chapman  and  Hall, 
London,  1966. 

The  foundational  text  on  analyzing  stochastic  point  processes. 

45.  T.  D.  Cox,  J.  Keely,  “Reliability  Growth  Management  of  SATCOM  Terminals,”  Proc. 
197(1  Annual  Reliability  and  Maintainability  Symp.,  1976,  pp  218-221. 

Comparison  of  Duane’s  model  to  Crow’s  nonhomogeneous  Poisson  process 
model  using  electronic  system  failure  data. 

46.  W.  P.  Cougan,  W.  G.  Kindig,  “A  Real  Life  MTBF  Program  for  a  Deployed  Radar,”  Proc. 
1979  Annual  Reliability  and  Maintainability  Symp.,  1979,  pp  212-127. 

Application  of  Duane’s  model  to  electronic  system  reliability  growth. 

47.  L.  H.  Crow,  Interim  Note  No.  27  On  Reliability  Growth  Modeling,  U.  S.  Army  Material 
Systems  Analysis  Activity,  Aberdeen  Proving  Ground,  MD,  Jan  1974. 

48.  L.  H.  Crow,  “Reliability  Analysis  for  Complex  Repairable  Systems,”  Soc.  Industrial  and 
Applied  Mathematics,  Reliability  and  Biometry,  Proceedings  of  Statistical  Analysis  of  Life 
Length,  Jul  1974,  pp  379-410 

Provides  derivation  of  maximum  likelihood  estimators,  hypotheses  testing, 
confidence  intervals,  and  goodness-of-fit  procedures  for  modeling  repairable 
system  reliability  growth  via  a  nonhomogeneous  Poisson  process  with  a  Weibull 
intensity  function. 

49.  L.  H.  Crow,  “On  Tracking  Reliability  Growth,”  Proc.  1975  Annual  Reliability  and 
Maintainability  Symp.,  1975,  pp  438-443. 

Application  of  Crow’s  model  to  missile  system  data. 

50.  L.  H.  Crow,  Confidence  Interval  Procedures  for  Reliability  Growth  Analysis,  U.S.  Army 
Material  Systems  Analysis  Activity,  Tech.  Rept.  197,  Aberdeen  Proving  Ground,  MD, 
Jun  1977. 

51.  L.  H.  Crow,  “Confidence  Interval  Procedures  for  the  Weibull  Process  With  Applications  to 
Reliability  Growth,”  Technometrics,  Vol  24,  Feb  1982,  pp  67-72. 


52.  L.  H.  Crow,  “An  Improved  Methodology  for  Reliability  Growth  Projections,”  Technical 
Report  357,  U.S.  Army  Material  Systems  Analysis  Activity,  Aberdeen,  MD,  June  1982. 

53.  F.  Croxton,  D.  Cowdwn,  General  Applied  Statistics,  Prentice-Hall,  Englewood  Cliffs,  NJ, 
Chap  13,  pp  282-319, 1956. 

Details  manual  computing  methods  to  estimate  the  parameters  of  the  Gompertz 
and  logistic  functions. 

54.  R.  C.  Dahiya,  “Estimation  of  Reliability  After  Corrective  Action,”  IEEE  Trans.  Reliability, 
Vol  R-26  Dec  1977,  pp  348-351. 

Gives  asymptotic  distributions  for  estimators. 

55.  B.  S.  Dhillon,  “Mechanical  Reliability  Interference  Theory  Models,”  Proc.  1980  Annual 
Reliability  and  Maintainability  Symp.  1980,  pp  462-467. 

Reliability  dealt  with  in  terms  of  stress  and  strength  concepts.  Large  bibliogra¬ 
phy. 

56.  B.  S.  Dhillon,  “New  Hazard  Rate  Functions,”  Microelectronics  and  Reliability,  Vol  18, 

1979,  pp  531-532. 

57.  B.  S.  Dhillon,  “Reliability  Growth:  A  Survey,”  Microelectronics  and  Reliability,  Vol  20, 

1980,  pp  743-751. 

Very  large  bibliography. 

58.  B.  S.  Dhillon,  “Life  Distribution,”  IEEE  Trans.  Reliability,  Vol  R-30,  Dec  1981,  pp  457-460 

Presents  two  new  distributions. 

59.  Y.  M.  I.  Dirickx,  K.  P.  Kistner,  “Reliability  of  a  Repairable  System  with  Redundant  units 
and  Preventive  Maintenance,”  IEEE  Trans.  Reliability,  Vol  R-28,  Jun  1979,  pp  170-171. 

60.  D.  A.  Dobbins,  “The  Dangers  of  Relying  on  Industry  as  a  Partner  in  Material 
Development,”  Test,  Aug/Sept  1980,  pp  8-13. 

61.  J.  Donelson  III,  “Cost  Model  for  Testing  Program  Based  on  Nonhomogeneous  Poisson 
Failure  Model,  ”  IEEE  Trans.  Reliability,  Vol  R-26,  Aug  1977,  pp  189-194. 

62.  J.  T.  Duane,  “Learning  Curve  Approach  to  Reliability  Monitoring,”  IEEE  Trans. 
Aerospace,  Vol,  2  Apr  1964,  pp  563-566. 

Duane’s  presentation  of  his  model. 

63.  D.  R.  Earles,  “Reliability  Growth  Prediction  During  the  Initial  Design  Analysis,”  Proc. 
IEEE  7th  Natl.  Symp.  on  Reliability  and  Quality  Control,  1961,  pp  380-393. 


Early  empirical  evaluation  of  reliability  growth. 
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64.  R.  G.  Easterling,  “Approximate  Confidence  Limits  for  System  Reliability,”  J.  Amer.  Stat. 
Assoc.,  Vol  67,  Mar  1972,  pp  220-222. 

65.  J.  D.  Ekings,  R.  L.  Sweetland,  "Burn-in  Forever?  There  Must  be  a  Better  Way,”  Proc. 
1978  Annual  Reliability  and  Maintainability  Symp.,  1978,  pp  286-293. 

66.  E.  A.  Elsayed,  A.  Zebib,  “A  Repairable  Multistate  Device,”  IEEE  Trans.  Reliability, 
Vol  R-28,  Apr  1979,  pp  81-82. 

67.  R.  A.  Evans,  “Models  for  Reliability  of  Repaired  Equipment,”  IEEE  Trans.  Reliability, 
Vol  19,  Nov  1970,  p  127. 

68.  B.  V.  Faden,  D.  Rodbard,  EXPFIT:  Exponential  Data  Processing,  Biomedical  Computing 
Technology  Information  Center  Computer  Code  Collection,  Vanderbilt  Medical  Center, 
TN„  1975. 

FORTRAN  code  to  fit  one,  two  and  three  component  exponential  functions. 

69.  K.  W.  Fertig,  V.  K.  Murphy,  “Models  for  Reliability  Growth  During  Burn-in:  Theory  and 
Applications,”  Proc.  1978  Annual  Reliability  and  Maintainability  Symp.,  1978,  pp  504-509. 

Presents  three  new  Bayesian  models  for  electronic  systems. 

70.  R.  W.  Fink,  “Screening  for  Reliability  Growth,”  Proc.  1971  Annual  Symp.  on  Reliability, 
1971,  pp  316-320. 

71.  J.  M.  Finkelstcin,  “Starting  and  Limiting  Values  for  Reliability  Growth,”  IEEE  Trans. 
Reliability,  Vol  R-28,  Jun  1979,  pp  111-114. 

Expands  Duane’s/Crow’s  model(s)  to  three  parameters  to  produce  mathematical 
limits  of  growth. 

72.  J.  M.  Finkelstein,  “Confidence  Bounds  on  the  Parameters  of  the  Weibull  Process,” 
Technometrics,  Vol  18,  No.  1, 1976. 

73.  E.  L.  Frome,  M.  H.  Kutner,  J.  J.  Beauchamp,  “Regression  Analysis  of  Poisson-Distributed 
Data,”  J.  Amer.  Stat.  Assoc.,  Vol  68,  Dec  1973,  pp  935-940. 

74.  W.  A.  Fuller,  Introduction  to  Statistical  Time  Series,  John  Wiley  &  Sons,  New  York,  NY 
1976,  pp  372-373. 

75.  A.  E.  Gelfand,  “An  Estimation  Problem  with  Poisson  Processes,”  Austral.  J.  Stat.,  Vol  23, 
Aug  1981,  pp  224-231. 

Reviews  Poisson  process  intensity  function  estimation. 
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76.  W.  R.  Gjertsen,  "Some  Steady  State  Queueing  System  Macros,”  Statistical  Analysis 
System  Users  Group  Internatl.  Proc.  of  the  6th  Ann.  Conf.,  SAS  Institute,  Cary,  NC,  1981, 
pp  304-308. 

Provides  SAS  code  to  test  for  exponential  interarrival  times. 

77.  A.  L.  Goel,  K.  Okumoto,  “Time  Dependent  Error-Detection  Rate  Model  for  Software 
Reliability  and  Other  Performance  Measures,”  IEEE  Trans.  Reliability,  Vol  R-28, 
Aug  1979,  pp  206-211. 

Presents  various  intensity  functions  for  nonhomogeneous  Poisson  processes 

applicable  to  general  reliability  growth. 

78.  0.  Gokcek,  I.  Bazovsky  Sr.,  G.  L.  Crellin,  “Markov  Analysis  of  Nuclear  Plant  Failure 
Dependencies,”  Proc.  1979  Annual  Reliability  and  Maintainability  Symp.,  1979,  pp  104-109. 
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APPENDIX  B 

Part  1  —  Criteria  for  Reliability-Growth  Model  Selection  as  Established  in  Task  200 

(July  1982) 

1.  Small  sample  forecasting  —  The  reliability-growth  can  be  predicted  for 
small  samples.  (5)* 

2.  Tracking  ability  —  The  reliability-growth  can  be  tracked  statistically,  i.  e., 
the  actual  growth  follows  the  predicted  growth  curves.  (6)* 

3.  Realistic  final  results  —  The  model  will  predict  final  results  that  are 
comparable  with  observed  data.  (2)* 

4.  Physical  interpretation  —  The  model  will  be  based  on  engineering  concepts 
as  well  as  statistical  concepts.  (7, 8)* 

5.  Graphic  display  of  results  —  Some  graphic  techniques  for  displaying  data 
and  model  will  be  available.  (9)* 

6.  Adaptability  —  The  model  will  be  able  to  handle  development  and  field 
data,  to  capture  the  check-mark  phenomenon,  and  to  handle  repairable 
systems.  (10, 11)* 

7.  Valid  goodness-to-fit  tests  —  Some  statistically  valid  goodness-to-fit  tests 
will  be  available  for  checking  the  reliability-growth  models.  (12)* 

8.  Realistic  asymptotic  values  — -  The  reliability-growth  model  will  produce 
realistic  asymptotic  values  with  the  increased  time  or  sample  size.  (14)* 

9.  Efficient/consistent  parameter  estimation  —  The  reliability-growth  model 
will  provide  means  of  accurate  and  consistent  estimation  of  various 
parameters .  (15)* 

10.  Confidence  limits  for  parameters  —  Methodology  will  be  available  to 
provide  interval  estimates  for  the  parameters  of  the  reliability-growth 
model.  (13)* 

11.  Test  for  trends  —  A  test  will  be  available  to  distinguish  between  random 
variation  and  actual  reliability  growth.  (13)* 

Part  2  —  Statements  Describing  Criterion  for  GAPCEEC  Model  Comparison 

1.  The  model  satisfactorily  generates  12-month  forecasts.** 

2.  The  model  satisfactorily  forecasts  reliability  at  maturity  from  development 
data. 

3.  The  model  forecasts  field  entry  reliability  within  ±  10%  with  adequate  lead 
time  for  review  and  action.** 

Corresponding  statement  number  in  Part  2. 

"Added  statements  to  include  work  done  on  long-term  forecasting. 


75 


**•»*«»?*•*  “  *»  r  - 

.*  *> * o o -  -  nr 


4.  The  model  forecasts  mature  life  reliability  within  ±  10%  with  adequate  lead 
time  for  review  and  action.** 

5.  The  model  generates  useful  forecasts  from  small  samples. 

6.  The  model  is  helpful  in  evaluating  reliability  growth  management  strategy. 

7.  The  models  parameters  can  be  equated  directly  to  reliability  engineering 
concepts. 

8.  The  model  has  meaning  in  a  physical  environment. 

9.  The  model  parameters  can  be  estimated  easily  with  graphic  displays. 

10.  The  model  can  adapt  quickly  to  changes  in  data  trends. 

11.  The  model  is  applicable  to  several  types  of  components  (electronic, 
mechanical,  etc.). 

12.  One  or  more  tests  are  available  that  measure  the  goodness-to-fit  between 
the  model  and  the  data. 

13.  One  or  more  tests  are  available  that  measure  the  contribution  of  the 
parameters  to  the  model’s  fit  of  the  data. 

14.  The  model  contains  a  limiting  parameter  that  limits  reliability  forecasts  to 
realistic  values. 

15.  Efficient,  consistent  estimation  of  the  models  is  clone  easily. 


APPENDIX  C 

MAXIMUM  LIKELIHOOD  ESTIMATION  OP 
PARAMETERS  OF  RELIABILITY  GROWTH  MODELS 


The  mathematics  of  the  maximum  likelihood  estimation  of  parameters  of  three  reliability 
growth  models  is  presented  in  this  appendix  (Duane,  Cox-Lewis  and  Modified  Duane  Models). 
The  Maximum  Likelihood  (ML)  estimators  of  the  Duane  Model  has  been  used  in  this  study. 

Maximum  Likelihood  Estimation  of  Parameters  of  Reliability  Growth  Models 

A.  BACKGROUND/GENERAL  INFORMATION 

Consider  a  nonhomogeneous  Poisson  process  with  intensity  function  p(t).  The  mean  value 
function  (expected  number  of  failures  by  time  t)  is 

M(t)  =  /I,  p  (x)  dx. 


The  probability  of  j  failures  by  time  t  (i.e.,  the  interval  o,t)  is 
P  0  Failures  e  (o,t))  =  f  M<° 


If  there  has  been  recorded  times  for  k  intervals,  i  =  1, . . .,  k  with  n;  failures  in  interval  i,  etc., 
then  the  probability  of  this  event  (nj  failures  in  interval  i)  is  as  follows: 


P(ns  failures  e  (t,_„  t,)) 


[M(t.)  -  M(tM)fe 

— r 


-  |M(t,>  -  MU,.  ,)| 


The  probability  of  all  intervals  having  the  observed  number  of  failures  is  the  product  of  the 
probabilities  associated  with  each  interval  (assuming  independence). 

k 

P(n,  failures  e  (o,t,)  and  n,  failures  e  (t„  t,)-  •  •  and  nk  failure  e  (tk  ,  tk))  =  n  P(n,  e  (t,_i,  ts)). 

i  -  ! 


This  is  the  likelihood  of  observing  exactly  what  has  happened  to  date,  and  is  called  the 
likelihood  function  (LF). 


the  (ML)  Maximum  likelihood  estimates  of  the  parameters  in  p  (t)  or  M(t))  are  those  parameter 
values  that  maximize  the  likelihood  function. 


LF  =  n  P(n,  8  (tM,  t,)) 


k  Nt.)  -  M(t._,)J- 

7i  - - — r~r - e 


1 


n, 


It  is  sufficient  to  maximize  the  loge  (LF).  (See  “Introduction  to  the  Theory  of  Statistics,” 
Mood,  Graybill,  and  Boes,  Third  Edition,  page  279.) 

log,'  (LF)  =  t  n;  log..  (M(t,)  -  M(t,  ,))  (M(t;)  -  M(t,  ,))  -  log,,  (n.) 


Since  logj  (nj)  does  not  depend  on  the  parameters  of  M(t),  it  is  sufficient  to  maximize  log*  (L'), 
where  loge  (L')  is  as  follows: 

loge(L')  -  I  n,  log,(M(t,)  -  M(tj.,))  -  (M(tj)  -  MM) 


which  can  be  simplified  to  the  following: 


log,(L')  -  — [M(tk)  -  M(t„)]  +  I  n,  log,  (M(tj)  -  M(tM)) 


B.  THE  LIKELIHOOD  EQUATIONS 

To  find  the  parameters  of  M(t)  that  maximize  log,.  (L'),  take  the  derivative  of  log,,  (L')  with 
respect  to  each  parameter,  set  the  equation  equal  to  zero,  and  solve.  If  M(t)  has  parameters  0j, 
0„ . . .  ©M,  then  the  set  of  equations  would  be  as  follows: 


8  log,  (L')  -8M(tk)  SM(t„)  ^ 

—wr~  “  +  +  z 


r  8M(tl)  8M(tj.,)  i 

k  niL  ~w - ran! 


to -mm 


81og,(L') 


r  8M(t,)  SMjt,,,)! 
-8M(tk)  8M(t(I)  £  "‘L  8©m"  J 

SBT  +  “W  +  t  M(tO  -  MCC.1 


Note  that  M  (t0)  =  0  for  all  models,  i.e.,  there  are  0  failures  by  time  t  -  0. 
C.  THE  DUANE  MODEL  (SEE  MIL-HDBK-189) 

p(t)  =  JiptP-  *■  X  and  P  are  parameters,  t  is  cumulative  time. 
M(t)  =  XtP 

=  tl>  and  -  Xt"  log,  (t). 

81oge(L)  lt  ^  ni  (tj  ~  L  i)  ,|i  ,/i  j  -  pp 

81og,(L')  n  N 

if  £  n,  =  N  then  — —  =  -tk  +  =  0.0 

l*  I 

51ogt,(L')  -  ..  ^  n;  [X  tf  logr(t,Htf  |  log,.(t,  ,)j 

5P - l0&-  W  +  "  1  _  1  *i« 


X  tf  -  X  t?_ , 


.  ,, .  ,n  £  ni  [*f log.(ti)  -  tf.,  log,(tM)] 

=  -Xt'l  log,(tk)  +  L - — jjj -  =  0.0. 


*> 
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Since  -t]J  +  -y-  =  0,  X  =  -^-.  Substituting  above, 

tk 

■^1  -  -f  •  4  MU  ♦  i  .  „.o 

.  _M  .  hrtj  ♦  i  "'[tflo^-^logM.  .  „.o 

H  >•!  tj  -  tj_, 

Which  can  be  solved  numerically  (with  a  computer)  for  p  »  p.  then,  using  P, 
obtained  from  the  equation 

a  A 


A  A 

X  =  N/tg. 


Therefore,  the  MLE’s  for  the  Duane  Model  are  P  and  X. 


D.  COX-LEWIS  MODEL 


p(t)  =  eY  '  ol  and  M(t)  =  —  eT  (e,a  -  1). 

-  l/aeT  (e"'  -  1). 

=  1/a  eT  [eal  (t  -  l/a)  -t  1/a]. 

81ofe(L/)  =  _i/aeV.«k  _  i )  +  i  n‘  [l/aeT(eat‘  -  1)  -  l/ae^e"1-1  -  1)] 
[  i-i  l/aeT  (eul‘  -  1)  -  l/ae^e01'-'  -  1) 

-  -  l/aer(e"'k  - 1)  +  N  =  0. 

S1°&S~  =  l/aer[e“,k(tk  -  1/a)  +  1/a]  + 

^  n,  l/aey  [e"'  (t,  -  1/a)  +  1/aj  -  l/aer[e'"'  '(t,  ,  -  l/a)  +  1/a]  ] 

.-i  l/aer  (e1"' -  1)  -  l/aeT  (e'"- 1  -  1) 


51°ga'----  =  -  l/a  e7  [eulk(tk  -  l/a)  +  l/a]  + 


f  n.  [(t.  "  l/a)  e<“‘  -  (ts- .  “  Va)  e<-,  ,J 

Z  =  00 

Since  — =  -  l/a  eY  (eolk  -  1)  +  N  =  0, 


-  l/aer  =  -  N/(e"'k  -  1). 


»  » .%  -+J.  »  <w« iJTtt  i-TV ^k.T>-«_^»  i «_7a- uZ^-—  #»~TI «LZ3U>^Ui»4^3ti 


JJ^AL.»jJ\K.W.li.Bj>-.a.li^)  £J^J.  JJ3itnr-i^L  .«1J1.-Mnjxiim  vJ.’ti.r  j  n  jyyy; 


Therefore,  a  can  be  estimated  by  the  following  equation: 

[e'^-l/a)  +  1/a]  +  Sn,[  -1/a]  -  0.0. 


(e"‘-l) 


Then  estimate  y  by 


y  -  log,  (N)  +  log,  (a)  -  log,  (ea,k  -  1). 


Thus,  the  MLE’s  for  the  Cox-Lewis  Model  are  a  and  y.  It  should  be  noted  that  there  are 
numerical  problems  in  estimating  a  and  y. 


E.  THE  MODIFIED  DUANE  MODEL 

p(t)  =  Xpt*1'  1  +  0.  X,  P,  and  0  are  parameters,  t  is  cumulative  test  time. 
M(t)=5ltl*  +  0t. 

i^tL  =  t,A^!L  .  *p.  .  wnorit) 


51og,(L')  i  ni  (ts  - 1,.,) 

-  “tk  +  L, 


T0 

Slog,(L') 


“SI 

81og..(L') 


-t!!  +  I 


Xtf  +  0t;  ~  Xtf_,  ~  0t,_, 
k  ni  (tf“  tf_j) 


0.0 


-1  Xtf  +  0t;  -  A.tf_,  -  Oti 


0.0 


,  V  ni  Wlo&  (t|)  -  **?-!  log,(ti-,)] 

-  =  -t£  log,  (tk)  +  L  — — - — - — - — -  *=  O.o. 


1-1 


Xtf  +  ©t.  -  Xtf ,  -  ©tj_, 


A  A  A 

Estimates  of  P,  0  and  X,  namely  p,  0,  and  X  can  be  obtained  by  solving  the  three  equations  above. 
This  is  a  difficult  problem  since  all  three  equations  are  nonlinear  with  respect  to  the  parameters. 
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APPENDIX  D 

TASK  300  —  PROCEDURE 


A.  PRELIMINARY  ANALYSIS/BACKGROUND  RESEARCH 
1.  Introduction 


This  reliability-growth  modeling  procedure  has  been  developed  to  assist  the  user  in 
applying  reliability-growth  modeling  technique  to  the  development  of  electronic  engine  controls 
(EEC).  The  primary  reference  used  in  developing  this  procedure  was  MIL-HDBK-189, 
“Reliability  Growth  Management.”  This  procedure  is  intended  to  serve  as  a  guide  for  applying 
the  AMSAA/Duane  model  (Army  Material  Systems  Analysis  Activity)  to  a  development 
program  and  individual  tailoring  should  be  expected.  It  is  assumed  that  reliability  has  been  a  goal 
of  the  development  program  and  was  considered  in  the  initial  design  of  the  system  and  in  the 
development  of  the  initial  test  plan.  A  typical  development  plan  is  illustrated  in  Figure  D-l. 
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Figure  D-l.  Typical  Development  Program 


2.  Review  of  the  Program  Plan 

It  is  important  to  understand  exactly  what  the  reliability  goals  are  and  if  they  are 
obtainable.  The  final  MTBF  (mean  time  between  failures),  will  be  needed  for  constructing  the 
planned  growth  rate. 

The  development  program  should  be  reviewed  in  detail  to  define  major  test  phases.  These 
test  phases  will  be  modeled  separately.  A  test  phase  is  a  distinct  period  of  time  during 
development  when  the  system  is  subjected  to  development  testing  and  subsequent  fixes  are  made. 
Test  phases  are  usually  aligned  with  management  goals  and  objectives.  A  test  phase  would 
usually  end  at  a  point  when  the  program  is  assessed  and  delayed  fixes  incorporated. 


For  example,  the  program  plan  shown  previously  in  Figure  D-l  could  initially  be  separated 
into  phases  as  indicated  in  Figure  D-2.  Phase  1  consists  of  the  initial  design  configuration 
(generation  1)  tested  on  engine  test  during  the  first  year.  This  phase  should  also  include 
additional  engine  and  flight  test  data  accumulated  on  generation  1  controls  (years  3  and  4).  After 
the  first  year,  a  second  generation  control  system  with  delayed  fixes  from  the  first  generation 
program  will  go  to  test.  This  test  is  designed  to  simulate  actual  operating  environment  and  the 
primary  purpose  of  the  test  is  to  develop  and  assess  the  reliability  of  the  control.  This  combined 
environment  reliability  test  (CERT)  is  considered  Phase  2.  Phase  3  corresponds  to  engine  and 
flight  test  data  from  generation  2  units.  Phase  2  and  3  data  can  possibly  be  pooled  since  they  are 
all  from  generation  2  units  but  this  should  be  done  only  after  reviewing  the  data.  Phases  4  and  5 
are  the  same  as  2  and  3,  but  for  generation  3  units. 
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Figure  D-2.  Typical  Development  Program  With  Test  Phases 


Each  of  the  defined  test  phases  should  be  reviewed  to  determine  their  expected  and  planned 
effect  on  reliability.  For  example,  phase  2  (CERT  on  generation  2  units  should  show  significant 
reliability  growth  because  of  the  ability  of  this  test  to  accumulate  large  amounts  of  time 
(50,000  hours)  within  a  short  calendar  period  (IV2  years). 

3.  Research  Historical  Information 


After  reviewing  the  test  plan  and  defining  test  phases,  the  background  information  on  this 
or  similar  systems  should  be  compiled.  Any  growth  modeling  information  on  this  or  similar 
systems  should  be  reviewed.  A  review  of  the  past  growth  rate  experience  and  initial  reliability  are 
also  important  when  considering  the  feasibility  of  a  planned  growth  curve.  A  review  of  the 
physical  makeup  of  the  system  would  also  be  helpful.  For  example,  a  mechanical  system  may  be 
expected  to  experience  more  early  wearout  failure  modes  than  an  electronic  system. 
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4.  Assessment  Capabilities 


Computer  power,  graphics  capabilities,  computer  software  available  and  data  systems  are 
important  considerations  in  model  selection  and  data  collection.  For  example,  Box-Jenkins  Time 
Series  Models  require  more  sophisticated  software  than  the  AMSAA/Duane  Model. 

5.  Data  Constraints 

A  data  collection  system  should  be  set  up  so  that  data  could  be  tracked  continuously.  While 
only  cumulative  fleet  time  at  the  time  of  failure  is  required  for  the  AMSAA  model  parameters  to 
be  estimated,  individual  component  times  and  the  cause  of  the  failure  are  required  when 
calculating  the  influence  of  delayed  fixes  on  reliability.  Therefore,  the  following  data  should  be 
collected: 

(1)  Serial  number  of  the  failed  unit 

(2)  Date  the  failure  occurred 

(3)  Time  on  the  failed  part 

(4)  Time  on  the  fleet 

(5)  Cause  of  the  failure. 

EEC  CERT  data  is  an  example  of  the  type  of  information  required. 

Data  is  usually  tracked  in  one  of  two  ways.  If  the  exact  amount  of  time  on  the  system  and 
fleet  is  known  for  each  failure,  the  system  is  being  tracked  continuously.  If  the  test  is  stopped 
after  a  prespecified  amount  of  time,  the  test  is  time  truncated.  For  example,  the  CERT  test  in 
Figure  1  is  planned  for  50,000  hours.  This  is  a  time  truncated  continuously  tracked  test. 
Sometimes  data  is  gathered  only  at  discrete  intervals.  For  example,  time  and  failures  may  be 
reported  by  month,  typical  of  field  data,  or  the  system  may  be  such  that  failures  are  found  at 
inspections.  This  data  is  group  data.  Continuously  tracked  data  is  always  preferred  when 
available. 

B.  MODEL  SELECTION 

There  are  many  models  available  for  use  in  tracking  and  predicting  reliability-growth 
(MIL-HDBK-189,  pp.  108-120).  Of  these  models,  the  AMSAA/Duane  model,  Box-Jenkins  Time 
Series  Modeling  and  the  Endless-Burn-In  model  are  good  candidates  for  modeling  the  reliability- 
growth  of  electronic  engine  controls. 

1.  The  AMSAA/Duane  Model 

The  AMSAA/Duane  Modeling  procedure  developed  by  AMSAA  personnel,  Aberdeen 
Proving  Grounds,  Maryland,  is  the  model  that  is  generally  recommended  for  use  in  modeling 
reliability-growth  of  an  electronic  engine  control  over  a  development  program. 

1'he  Duane  model  is  used  for  planning  the  general  reliability  growth  across  the  entire 
development  program  (all  test  phases).  This  is  a  management  tool  that  is  used  'o  determine  the 
feasibility  of  the  program  and  for  comparing  reliability  at  various  points  throughout  the  program. 

The  AMSAA  model  is  a  statistical  tool  useful  for  short  term  projections  of  reliability, 
assessment  of  current  reliability  and  comparison  of  present  levels  of  reliability  to  the  reliability 
goals. 
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a)  Model  Description 


The  Duane  model  is  a  curve  of  cumulative  failure  rate  as  a  function  of  cumulative  test  time 
in  the  development  program. 

C(t)  =  yta  where  C  (t)  is  cumulative  failure  rate  at  times  t 
Y  is  the  scale  parameter 
a  is  the  growth  parameter 

Taking  the  log  of  both  sides  of  the  equation,  the  model  becomes  linear  on  log-log  paper  with 
slope  a. 

log  (C(t))  =  log  (y)  +  a  log  (t) 

The  AMSAA  model  is  a  statistical  model  of  the  same  form  as  the  Duane  model.  It  is 
different  in  that  it  is  modeling  the  reliability  within  a  test  phase  as  a  function  of  time 
accumulated  in  the  test  phase. 

N(t)  =  Xt^  N(t)  is  cumulative  failures  by  time  t 
X  is  the  scale  parameter 
P  is  the  growth  parameter 

b)  Model  Assumptions 

The  Duane  model  assumes  that  the  cumulative  failure  rate  grows  along  the  curve  yt“  as  a 
function  of  development  time. 

The  AMSAA  model  assumes  that  the  failures  are  from  a  nonhomogeneous  poisson  process 
(NHPP)  with  a  Weibull  intensity  function.  The  intensity  function  is  as  follows: 

p(t)  =  Xptf}~1 

It  is  further  assumed  that  if  no  further  effort  is  made  to  improve  the  reliability,  the  failure 
rate  will  remain  constant  (time  between  failures  will  follow  an  exponential  distribution).  The 
following  assumptions  are  a  result  of  the  NHPP  assumption. 

•  N(0)  =  0  (no  failure  can  occur  without  some  test  time) 

•  The  probability  of  a  failure  at  time  t  is  not  affected  by  previous  failures 

•  Simultaneous  failures  are  not  possible. 

2.  Box-Jenkins  Time  Series  Modeling 

This  modeling  procedure  is  a  very  good  alternative  to  the  AMSAA/Duane  model  when  the 
quality  of  data  is  poor,  or  the  AMSAA  model  assumptions  are  in  question.  This  procedure  does 
not  require  that  a  specific  model  form  be  selected  i  advance,  however,  a  first  difference  moving 
average  model  is  generally  accepted.  The  Box-Jenkins  methodology  has  a  built-in  theory  of 
forecasting  and  generally  does  as  well  as  the  AMSAA  model  in  making  short  term  predictions  (up 
to  12  months)  of  reliability. 


a)  Model  Description 

The  Box-Jenkins  ARIMA  (Auto-Regressive-Integrated-Moving-Average)  model  is  a  model 
of  reliability  against  calendar  time.  The  function  that  defines  cumulative  failure  rate  at  a  given 
point  in  calendar  time  (A.cT)  is  a  function  of  previous  failure  rates,  random  errors  and 
parameters.  A  first  difference  moving  average  model  is  recommended,  as  follows: 

^■c.T ~%+  \,t~i  +  &r  —  6i  &T-1 
or 

^c,T  ~  'Vt-1  “  ®0  +  “  ®1  ®T-1* 

0O  is  the  trend  parameter.  This  parameter  may  frequently  be  insignificant  and  therefore  not 
included.  Statistical  tests  are  available  (Appendix  E)  for  determining  if  it  is  significant. 

Act  ~  ^c,T-i  *s  the  first  difference  part  of  this  first  difference  moving  average 
and 

&P  -  0j  &!>_!  is  a  first  order  weighted  (0! )  moving  average  of  the  random  error 
terms. 

Occasionally  0j  will  not  be  significant  and  the  model  will  be  a  simple  first  difference  model  or 
sometimes  referred  to  as  a  random  walk. 

^c,T=^c,T-l  +  eT 

b)  Model  Assumptions 

Box-Jenkins  Time  Series  Modeling  assumes  that  the  residual  effect  or  variation  unex¬ 
plained  by  the  model  is  normally  distributed  with  mean  0  and  variance  aa2.  It  is  further  assumed 
that  the  reliability  is  tracked  at  equally  spaced  time  intervals  (e.g.  monthly). 

3.  Endless-Bum-ln  Model 

This  model  takes  into  account  past  burn-in  and  uses  the  average  age  of  the  population 
rather  than  cumulative  age.  It  also  states  that  reliability  growth  approaches  a  limiting  value  (\). 
The  model  describes  instantaneous  failure  rate  as  a  function  of  average  test  time.  This  model  is 
intended  for  solid-state  electronic  failure  modes  only.  This  model  was  effective  in  making  short 
term  forecast  (e.g.  12  months)  and  the  parameters  were  easier  to  estimate  than  the  modified 
Duane  model  (another  3  parameter  model).  Honeywell  Avionics,  St.  Louis  Park,  Mo,  has  used 


the  model  successfully  on  the  Digital  Air  Data  Computer  (DADC)  for  use  on  the  DC-10  aircraft 
(See  Appendix  A  No.  23). 
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a)  Model  Description 


The  form  of  the  model  is  as  follows: 

X,-KT.ve"a  +  Xt 

Xj  is  the  instantaneous  failure  rate 

\  is  the  limiting  failures  rate  for  Xj 

K,  a  are  scale  and  growth  parameters,  respectively 

T#ve  is  the  average  fleet  time. 

b)  Model  Assumptions 

The  Endless-Burn-In  model  assumes  that  the  instantaneous  failure  rate  grows  along  the 
curve  K  T-a  +  \  as  a  function  of  average  age  or  operating  time.  This  assumption  implies  the 
following: 

•  Average  age  is  a  strictly  increasing  function  of  calendar  time.  This  could  be  a 
probbm  if  several  new  units  are  introduced  into  the  fleet  during  the 
development  program.  For  example,  in  Figure  D-l,  during  the  first  5  years  of 
development  18  units  accumulated  approximately  56,000  hours  for  an 
average  of  3111  hours.  When  the  third  generation  was  introduced  into  the 
program  in  the  fifth  year,  the  average  age  dropped  to  1168  hours 
(56,000/48  =>  1166)  and  by  the  end  of  the  program  climbed  back  to  1510. 

•  The  average  age  is  assumed  to  be  a  good  estimate  of  a  typical  unit  on  test. 

This  is  not  the  case  in  the  Figure  D-l  example,  since  some  units  accumulate 
large  amounts  of  time  (CERT  units)  and  others  do  not  accumulate  much 
time  (flight  test  units).  In  addition  to  the  average  age  assumption,  it  is 
assumed  that  an  adequate  amount  of  test  time  will  be  accumulated  to 
estimate  or  see  the  effect  of  the  limiting  parameters. 

An  example  of  a  program  where  these  assumptions  are  met  would  be  the  DADC  Honeywell 
data.  Here,  all  units  accumulated  time  at  about  the  same  rate  and  the  total  program  has 
accumulated  over  2  million  hours.  The  model  was  successfully  applied  to  the  electronics  failure 
modes  (see  the  article  “Growth  Modeling  Improves  Reliability  Predictions,”  by  Bezat,  Norquist 
and  Montague,  Annual  Reliab.  and  Maint.  Sym.,  Proc.  1975). 

C.  DESIGN  THE  IDEAL  GROWTH  CURVE  FOR  TRACKING 

In  designing  the  ideal  growth  curve,  tbf.  following  information  is  usually  required: 

•  Initial  mean  time  between  fai>  1  (MTBF)  (Mj).  This  is  usually  estimated 
from  some  initial  testing. 

•  The  amount  of  initial  testing  Tj. 

•  The  final  MTBF  (MF).  This  is  usually  the  reliability  goal  for  the  program. 

•  The  cumulative  amount  of  test  time  allocated  for  the  program  TF. 


The  model  being  fit  is  as  follows: 
M(t)  =  Mi(-7jjr-)“  (1-a).'1 


The  ideal  curve  has  a  base  value  of  M;  at  time  T^  M(t)  increases  from  Tj  to  TF  where  it 
reaches  the  value  M. 


To  approximate  in  the  equation  above  when  a  is  less  than  or  equal  to  0.5,  observe  the 
following: 


a  =  -log(^)-l+[(l  +  log(^)) 


+  2  log 


For  values  of  a  >  0.5  (very  optimistic  programs),  use  iterative  techniques  for  solving  for  a. 

NOTE:  If  an  additional  point  on  the  curve  is  known,  a  can  be  calculated  directly. 

Suppos' the  MTBF  at  the  time  tj  is  known  to  be  Mj,  then 

log(MF)  -  log  (M,) 

°  =  log  (Tf)  -  log  (t,)  • 

1.  Example  of  the  Calculation  of  the  Growth  Parameter 

Assume  that  an  initial  MTBF  of  100  hours  has  been  demonstrated  from  the  first  1000  hours 
of  generation  1  testing,  shown  previously  in  Figure  D-l.  There  is  a  total  ot  75,650  hours  planned 
for  this  development  program  and  a  3000  hour  MTBF  is  desired.  Note  the  following  formula: 


a  =  0.605. 


This  indicates  that  the  program  is  very  aggressive.  Also  note  that  for  a  >  0.5,  an  iterative 
procedure  is  recommended.  In  this  example,  a  computer  program  written  in  FORTRAN  and 
using  IMSL  (International  Mathematical  and  Statistical  Library)  was  developed  to  ensure  the 
accuracy  of  a.  This  program  shown  in  Figure  D-3  gave  an  a  =  0.59.  The  ideal  growth  curve  is  also 
illustrated  in  Figure  D-3.  Fig  ;re  D-4  shows  this  same  curve  on  log-log  scale. 


2.  Determine  the  Average  MTBF  for  Each  Test  Phase 

First,  compute  the  expected  total  failures  N(tj)  by  the  completion  of  phase  i,  at  time  t;. 

nw  =  -rar^f-tr) 
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The  average  MTBF  in  phase  i  is  as  fallows: 

MTBP  ■  TTH^fcr 


Phase 

t_ 

N_ 

MTBF 

1 

1,000 

10.0 

100 

2 

52,000 

50.5 

1009 

3 

55,000 

51.7 

2500 

4 

65,000 

55.4 

2702 

5 

75,650 

59.2 

2802 

Figure  D-5  shows  the  average  MTBF  sketched  over  the  ideal  jurve,  previously  shown  in 
Figure  D-3. 


Figure  D-5.  Ideal  Growth  Curve  for  Development  Plan  (With  Average  MTBF  for 
Each  Phase) 


3.  Evaluating  the  Feasibility  of  the  Ideal  Curve 

The  ideal  curve  should  be  compared  to  past  experience  to  determine  if  the  growth  within 
each  phase  is  reasonable.  It  should  be  noted  that  the  ideal  growth  curve  does  not  consider  delayed 
fixes.  Therefore  the  planned  curve  should  consider  delayed  fixes.  Figure  D-6  shows  that  at  some 
point  fixes  will  no  longer  be  implemented  until  the  end  of  the  test. 
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Figure  D-6.  Ideal  Growth  Curve  for  Development  Plan 

In  the  example,  a  growth  rate  a  =  0.59  is  considered  higher  than  usual  but  still  obtainable. 
The  rate  would  be  expected  to  remain  constant  until  the  fixes  are  implemented.  A  jump  would  be 
seen  at  the  start  of  the  next  phase  as  a  result  of  these  fixes. 

The  design  is  reviewed  to  determine  if  the  system  is  considered  an  evolution  of  previous 
controls  systems  or  a  system  that  is  considered  to  be  revolutionary.  Resources  should  also  be 
checked  to  ensure  that  the  goals  are  obtainable. 

D.  ANALYSIS  OF  DATA 

Once  the  program  is  underway  and  data  has  been  collected,  it  will  be  necessary  to  analyze 
the  data  for  purposes  of  assessment,  prediction  and  control.  This  analysis  involves  engineering 
and  statistical  methodology  with  examples  using  continuous  time  truncated  dc.ta  and  grouped 
data. 


1.  Objectives  of  the  Analysis 

The  primary  objectives  of  the  analysis  are  as  follows: 

•  Assessment  —  An  estimate  of  the  current  reliability  with  confidence  limit  (if 
possible)  should  be  made  to  assess  the  reliability. 

•  Prediction  —  A  forecast  of  future  re'dability  over  some  future  time  period 
would  be  helpful  for  planning  purposes. 

•  Control  —  A  comparison  of  the  demonstrated  reliability  (assessment)  to  the 
planned  reliability  is  helpful  in  determining  if  the  program  is  on  schedule, 
hence  under  control. 


2.  Engineering  Analysis  of  Failure  Data 


Engineering  analysis  of  failure  data  is  performed  to  (1)  understand  the  failure  modes  that 
exist  in  a  component  or  system,  (2)  evaluate  design  changes  to  eliminate  these  failure  modes,  and 
(3)  apply  this  information  to  the  reliability  growth  model  that  represents  the  component  or 
system.  This  analysis  is  of  particular  importance  where  design  changes  were  not  incorporated  in 
a  particular  design  phase  or  where  adequate  time  did  not  exist  to  verify  the  improvements. 

a)  Preliminary  Analysis 

Initial  analysis  of  the  failure  data  is  conducted  to  determine  completeness  and  accuracy.  It 
is  desirable  to  have  individual  unit  time  at  failure,  cumulative  fleet  time  at  failure,  and  detailed 
information  on  the  circumstances  surrounding  each  failure  event.  This  information  is  generally 
available  during  a  reliability  development  program  such  as  CERT,  as  discussed  in  Section  1.5. 
Field  data  is  often  less  detailed  and  based  on  group  data  (such  as  monthly).  This  information, 
although  not  as  complete  as  desired,  is  sufficient  to  do  growth  modeling,  provided  failure 
investigation  information  is  available. 

After  failure  data  has  been  collected  and  is  considered  complete,  it  must  be  reviewed  to 
ensure  that  all  reported  malfunctions  are  confirmed  to  be  component  failures.  In  both 
development  testing  and  field  operation,  the  reported  malfunction  of  a  component  is  often  found 
to  be  unconfirmed  or  induced  after  detailed  analysis  and  testing.  The  unconfirmed  failures  are 
often  due  to  poor  troubleshooting,  incomplete  procedures,  or  lack  of  diagnostic  equipment.  The 
induced  failures  are  caused  by  mishandling,  misoper.iticn,  failures  of  other  hardware,  or  use 
beyond  established  limits.  Because  these  unconfirmed  and  induced  failures  are  not  necessarily  a 
function  of  the  true  component  reliability,  these  reported  malfunctions  should  be  eliminated 
from  the  failure  data  prior  to  growth  modeling. 

b)  Failure  Mode  Assessment 

The  failure  data  should  be  further  analyzed  to  identify  failures  by  mode.  Each  failure  mode 
should  then  be  reviewed  to  ensure  that  sufficient  investigation  has  been  completed  to  understand 
the  reason  for  failure.  At  this  point  the  corrective  action  can  be  evaluated  as  to  its  effectiveness 
and  the  expected  impact  of  the  changes  on  reliability  growth  evaluated.  This  impact  on  growth  is 
influenced  by  the  accuracy  of  the  failure  mode  investigation,  the  effectiveness  of  corrective 
actions,  and  whether  that  corrective  action  is  implemented  during  or  after  the  testing  phase. 

Table  D- 1  shows  a  summary  of  EEC  CERT  data  to  be  used  as  an  example. 

For  each  failure  during  the  test,  the  malfunction  cause  was  identified,  each  malfunction  was 
investigated,  and  corrective  action  formulated,  as  shown  in  Table  D-l.  In  this  example  some 
corrective  actions  were  implemented  during  testing  and  some  were  implemented  as  delayed  fixes. 

The  failure  mode  identified  as  Socket  Damage  in  Table  D-l  was  investigated  and  found  to 
be  due  to  inadequate  process  controls  at  the  time  of  assembly.  Corrective  action  was  formulated 
and  all  units  retrofitted  using  the  new  process  controls.  No  further  malfunctions  were  noted  in 
over  45,000  hours  of  test,  resulting  in  reliability  growth  during  the  CERT  test  phase. 


TABLE  D-l.  EEC  CERT  FAILURE  SUMMARY 


Item 

Number 

MR 

Number 

Unit 

Number 

Fault 

Date 

UUT  Time 
to  Mali 

AU 

UUT  Time 
to  Malt. 

Malfunction 

Cause 

Failure 

Class. 

Corrective 

Action 

1 

E001 

6 

1/7/81 

2.0 

4.0 

Socket  Damage 

EMW 

Add  HS9575  Req’ts 

2 

E002 

3 

1/7/81 

10.4 

20.8 

TPT2.Interm.Unk. 

Other 

Monitor  and  TBD 

3 

E003 

2 

1/13/81 

21.9 

295.0 

Socket  Damage 

EMW 

Add  HS9575  Req’ta 

4 

E004 

5 

1/13/81 

22.5 

295.0 

Socket  Damage 

EMW 

Add  HS9575  Req’ta 

5 

E005 

4 

2/17/81 

628.0 

2459.0 

Socket  Damage 

EMW 

Add  HS9575  Req’ta 

6 

EOOf 

3 

3/4/81 

1070.0 

3100.0 

Socket  Damage 

EMW 

Add  HS9575  Req’ta 

7 

EOOt 

6 

5/12/81 

1351.1 

7274.0 

Incorrect  tool  use 

EMW 

Added  info  +  QC  test 

8 

E009 

2 

6/8/81 

1735.7 

8996.3 

Socket/lead  contact 

EMD 

Process  change 

9 

E012 

4 

8/18/81 

3466.0 

15522.1 

Socket/lead  contact 

EMD 

Process  change 

10 

E013 

O 

9/29/81 

4651.0 

19988.7 

Lead  abrasion 

EMD 

In  process,  TBD 

11 

E015 

2 

10/12/81 

3395.8 

20751.8 

Resistor  network 

PMD 

ECl4277o 

12 

EG07 

5 

10/25/81 

3619.8 

21840.6 

Socket/lead  contact 

EMD 

Process  change 

13 

£020 

2 

ii/.vsi 

3822.5 

22695.5 

Socket/lead  contact 

EMD 

Process  change 

14 

EOlo-1 

3 

11/3/31 

5176.9 

22793.2 

Resistor  network 

PMD 

EC  142775 

15 

E016-2 

3 

11/3/81 

5176.9 

22793.2 

Lead  abrasion 

EMD 

In  process,  TBD 

16 

E019 

4 

1/9/82 

4866.0 

26872.9 

Socket/lead  contact 

EMD 

Process  change 

17 

E021 

3 

2/24/82 

6407.8 

29684.0 

Socket/lead  contact 

EMD 

Process  change 

18 

E022 

2 

4/27/82 

5706.2 

33853.1 

Socket/lead  contact 

EMD 

Process  change 

19 

E023 

1 

5/17/82 

5473.5 

35342.5 

Loose  connector 

PMD 

EC140374 

20 

E0,5 

3 

6/1/82 

7672.3 

36082.7 

Resistor  network 

Other 

None,  part  damaged 

21 

E028 

1 

7/16/82 

6586.9 

39663.7 

Socket/lead  contact 

EMD 

Process  change 

22 

E029 

4 

8/4/82 

5366.0 

41662.7 

Socket/lead  contact 

EMD 

Process  change 

23 

E030 

5 

8/9/82 

5805.2 

42242.1 

Socket/lead  contact 

EMD 

Process  change 

24 

E031 

3 

8/13/82 

8867.1 

42586.2 

Prom,  bent  lead 

EMW 

Add  HS9575  Req’ta 

25 

E032 

2 

8/19/82 

7133.4 

42998.0 

Socket/ieid  contact 

EMD 

Process  change 

26 

E037 

5 

8/20/82 

5988.5 

43059.4 

Res.  WW,  crimp  are.' 

EMD 

None/isolated  case 

27 

E033 

4 

9/4/82 

5865.7 

45.003.0 

Socket/lead  contact 

EMD 

Process  change 

28 

£034 

1 

9/5/82 

7044.6 

45025.0 

Socket/lead  contact 

PMW 

Process  change 

29 

E040 

5 

10/18/82 

6809.4 

48507.8 

Socket/lead  contact 

EMD 

Process  change 

The  failure  mode  identified  as  Socket/ 1 ^ad  Contact  in  'Fable  D-l  was  investigated  and 
found  to  be  due  to  adhesive  contamination.  Corrective  action  was  identified  which  changed  the 
adhesive  and  eliminated  this  failure  mode.  Corrective  action  for  this  mode  however  was  not 
implemented  until  late  in  the  test.  Reliability  growth,  t  herefore,  would  not  be  expected  during  the 
test  as  a  result  of  this  delayed  fix.  Improvement  due  to  this  fix  is  calculated  using  the  techniques 
described  in  Section  4.3.6.  The  effectiveness  of  each  corrective  action  required  by  this  technique 
is  based  upon  engineering  judgement  using  all  available  information.  For  this  failure  mode,  the 
cause  was  well  established  by  detailed  failure  analysis.  The  corrective  action  will  eliminate  the 
primary  source  of  contamination  found  in  failed  units.  Testing  of  a  similar  design,  however, 
indicates  that  contamination  from  other  sources,  such  as  human  error  in  assembly  and  contact 
wear,  will  continue  to  occur.  A  corrective  action  effectiveness  for  this  problem  of  75%  results  in 
an  expected  rate  of  60/106  hours.  Engineering  evaluation  indicates  that  this  is  a  reasonable  rate 
for  the  next  phase  of  testing.  Some  additional  guidance  is  provided  in  MIL-HDBK-189 
Appendix  A  for  making  engineering  evaluations  for  delayed  fixes. 

Evaluation  of  field  data  requires  similar  techniques  as  described  by  this  example.  Data, 
however,  is  usually  group  data  and  fixes  tend  to  be  delayed. 


c)  Continued  Engineering  Analysis 


Engineering  analysis  of  the  drta  should  continue  throughout  the  statistical  analysis.  This 
includes  review  of  plots,  identification  of  engineering  reasons  for  any  discontinuities  observed 
and  estimating  the  effectiveness  of  all  delayed  fixes. 

3.  Statistical  Analysis 

Within  a  test  phase,  it  is  assumed  that  problems  are  being  identified  and  corrected  and,  as  a 
result,  the  reliability  of  the  system  is  improving.  Since  the  system  has  been  changed,  data  from 
the  earlier  part  of  the  test  is  not  representative  of  the  current  systems;  however,  there  is  usually  a 
very  limited  amount  of  current  data.  Because  of  this  situation,  reliability  growth  models  (such  as 
the  AMSAA/Duane  model)  are  employed.  These  models  make  use  of  the  combined  data  in 
modeling  the  growth  process  and  making  estimates  of  current  and  future  reliability  in  the 
presence  of  a  changing  configuration. 

a)  Preliminary  Analysis  of  Data 

Before  attempting  to  use  the  model  and  make  reliability  assessments,  it  is  important  to 
review  the  data  for  oroblems  such  as  outliers  and  missing  data.  The  best  approach  to  such 
problems  is  usually  to  simply  look  at  the  data.  Plots  of  cumulative  failures  vs  cumulative  time, 
average  failure  rate  vs  cumulative  time,  and  cumulative  MTBF  vs  cumulative  time  are  helpful  in 
identifying  problems  in  the  data  as  well  as  determining  the  usefulness  of  models. 

Cumulative  Failures  vs  Cumulative  Time 

This  plot  is  helpful  in  assessing  the  fit  of  the  AMSAA  model.  If  the  AMSAA  model  is 
appropriate,  the  data  should  be  linear  on  log-log  paper.  Recall  that  the  expected  total  failures  by 
time  T  is  N  =  XT*1 .  Taking  the  natural  logarithm  of  both  sides. 

log  (N)  =  log  (A)  +  P  log  (T). 

If  there  is  a  distinct  break  in  the  data,  it  may  indicate  a  change  in  emphasis  in  the  program, 
significant  change  in  technology  level,  or  an  unpredicted  change  in  the  testing  environment  as 
well  as  a  number  of  other  reasons.  It  should  be  reviewed  and  the  data  modeled  in  two  parts. 

For  example,  the  following  data  in  Table  D-2  is  from  a  typical  electronic  control  CERT  test 
which  would  correspond  to  the  second  generation  CERT  test  in  the  simulated  example  shown 
previously  in  Figures  D-l  and  -2. 

After  reviewing  the  data  for  obvious  problems  and  reviewing  the  engineering  analysis  to 
ensure  that  each  failure  is  applicable  to  the  program  and  not  caused  as  a  result  of  test  problems, 
etc.,  the  cumulative  failure  vs  cumulative  time  plot  is  produced  (See  Figure  D-7). 

From  this  plot  there  appears  to  be  a  break  at  about  20,000  hours.  During  the  first  20,000 
hours  the  growth  rate  was  very  high  (P  =  0.37,  or  in  terms  of  the  Duane  curve,  a  =  0.63).  The  last 
30,000  hours,  20,000  to  50,000,  showed  very  little  growth  (P  =  0.92  or  a  =  0.078).  Figure  D-8 
shows  the  first  20,000  hours  of  data  shown  in  Figure  D-7.  Figure  D-9  shows  the  last  30,000  hours 
of  data  shown  in  Figure  D-7. 
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TABLE  D-2.  EEC  CERT  TEST 


Incident * 

Unit 

Unit  'Time 

Fleet  Time 

1 

E001 

6 

2.0 

4.0 

2 

E002 

3 

10.4 

20.8 

3 

E003 

2 

21.9 

295.0 

4 

15004 

5 

22.5 

295.0 

5 

E005 

4 

628.0 

2459.0 

6 

E006 

3 

1070.0 

3100.0 

7 

E008 

6 

1351.1 

7274.0 

3 

E009 

2 

1735.7 

8996.3 

9 

E012 

4 

3466.0 

15522.1 

10 

E013 

3 

4651.0 

19988.7 

11 

E015 

2 

3395.8 

20751.8 

12 

E007 

5 

3619.8 

21840.6 

13 

E020 

2 

3822.5 

22695.5 

14 

E016-1 

3 

5176.9 

22793.2 

15 

E016-2 

3 

5176.9 

22793.2 

16 

E019 

4 

4866.0 

26872.9 

17 

E021 

3 

6407.8 

29684.0 

18 

E022 

2 

5706.2 

33853.1 

19 

E023 

1 

5473.6 

35342.5 

20 

E025 

3 

7672.3 

36082.7 

21 

E028 

1 

6586.9 

39663.7 

22 

E029 

4 

5366.0 

41662.7 

23 

E030 

5 

5805.2 

42242.1 

24 

E031 

3 

8867.1 

42586.2 

25 

E032 

2 

7133.4 

42998.0 

26 

E037 

5 

5988.5 

43059.4 

27 

E033 

4 

5865.7 

45003.0 

28 

B034 

1 

7044.6 

45025.0 

29 

E040 

5 

6809.4 

48507.8 

Unit 

Total  Time 

~r 

8,234.3"" 

2 

8,396.9 

3 

9,740.6 

4 

6,235.3 

5 

6,930.3 

6 

10,507.3 

50,044.9 

i 
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Figure  D-9.  Reliability  Growth  of  CERT  (Cumulative  Incidents  During  Last  30K 
Test  Hours) 

Average  Failure  Rate  Plots 

The  average  failure  rate  plot  shows  how  the  failure  rate  is  changing  with  time.  The  average 
failure  rate  is  computed  over  each  5,000  hour  interval  and  plotted.  Figure  D-10  shows  all 
50,000  hours.  Figure  D-ll  shows  the  first  20,000  hours  and  Figure  D-12  shows  the  last  30,000 
hours. 
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Figure  l)-10.  CERT  Analysis  Reliability  Growth  Analysis  Average  Failure  Rate  Plot 
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Cumulative  MTBF  vs  Cumulative  Time  Plots 

This  plot  illustrates  the  original  learning  curve  proposed  by  Duane  as  shown  in 
Figures  D-13,  D-14  and  D-15,  which  are  transformations  of  Figures  D-7,  D-8  and  D-9, 
respectively,  and  also  illustrates  the  20,000  hour  break. 
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Figure  D-13.  Reliability  Growth  of  50K  Hour  CERT  (Cumulative  MTBF  vs  Cumulative 
Test  Time) 
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Figure  D-14.  Reliability  Growth  of  CERT  (Cumulative  MTBF  During  First  20K 
Test  Hours) 
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Figure  D-15.  Reliability  Growth  of  CERT  (Cumulative  MTBF  During  Last  30K 
Test  Hours) 

Plots  Applied  to  Field  Data 

While  the  AMSAA  procedure  was  intended  for  use  within  phases  of  a  development 
program,  these  are  also  helpful  in  analyzing  data  from  field  experience  where  a  fleet  of  controls 
has  been  tracked,  reviewed  and  changes  made  for  improving  the  reliability  via  retrofit  programs. 
As  an  example,  listed  in  Table  D-3,  consider  the  EEC  JFC-90  used  on  the  F100  engine.  This  data 
is  also  illustrated  in  Figure  D-16. 

TABLE  D-3.  EEC  FIELD  DATA 


YEAR 

ONE  1 

YEAR 

TWO 

YEAR 

THREE 

YEAR 

FOUR 

YEAR 

FIVE 

YEAR 

SIX 

i 

YEAR 

EEVFN 

i 

CUM  TIME 

CUM  FAIL  1 

CUM  TIME 

CUM  FAIL 

CUM  TIME 

CUM  FAU 

CUM  TIME 

CUM  FAIL 

CUM  TIME 

CUM  FAIL 

CUM  TIME 

CUM  FAX  L 1 

CUM  TIME 

cm  fail! 

164.0 

1.0  | 

10852.8 

33.0 

65332.6 

122.0 

196974.1 

239.0 

413148.1 

412.0 

721624.6 

782.0 

1097796.0 

1319.0 

465.6 

1.0  1 

12866.4 

37.0 

73039.9 

127.0 

210527.7 

253.0 

434554.4 

435.0 

752443.6 

633.0 

1130756.0 

1360.0 

731. 2 

3.0  | 

15256.0 

41.0 

80839.9 

137.0 

223617.2 

264.0 

454564.6 

454.0 

761784.6 

669.0 

l 

1166280.0 

1400.0 

1310.4 

4.0  | 

16110.4 

50.0 

91043.1 

148.0 

241025.2 

276.0 

460746.0 

494.0 

612659.7 

900.0 

1207144.0 

1449.0 

1924.8 

9.0  I 

21870.4 

56.0 

100612.7 

157.0 

257230.0 

292.0 

507207.2 

524.0 

646161.3 

952.0 

1248801.0 

1510.0 

2662.4 

12.0  I 

25348.6 

65.0 

112110.2 

163.0 

275647.6 

306.0 

533764.7 

545.0 

675002.1 

985.0 

1287153.0 

1555.0 

3652.8 

14.0  | 

29414.4 

76.0 

123790.2 

179.0 

296004.3 

316.0 

559416.7 

581.0 

906968.4 

1032.0 

1331340.0 

1601.0 

1 

4500.8 

17.0  | 

34145.6 

64.0 

134331.0 

189.0 

312673.1 

336.0 

564322.3 

613.0 

936606.8 

1081.0 

1376561.0 

1653.0 

5532  8 

22.0  | 

39528.0 

95.0 

146750.2 

203.0 

335696.6 

354.0 

612727.1 

655.0 

967003.6 

1125.0 

1414571.0 

1700.0 

6500.8 

23.0  | 

45168.0 

106.0 

158350.2 

209.0 

353953.0 

373.0 

635467.1 

695.0 

996624.3 

11/2.0 

1451906.0 

1738.0 

8049.6 

25.0  | 

51062.4 

108.0 

170886.2 

220.0 

374845.7 

366.0 

666663.1 

722.0 

1035277.1 

1217.0 

1498540.0 

1781.0 

9102.4 

29.0  | 

58145.6 

114.0 

164127.7 

229.0 

394063.3 

398.0 

695765.4 

753.0 

1065329.0 

1261.0 

1S42906.0 

1826.0 

Several  breaks  are  apparent  here,  indicating  changes  in  the  system  or  operating  environment. 

Average  Failure  Rate/1000  hours  is  shown  in  Figure  D-17. 

Cumulative  MTBF  is  shown  in  Figures  D-18  and  D-19. 

The  data  suggests  that  the  failure  rate  has  been  relatively  constant  over  the  past  2  years. 
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b)  Parameter  Estimation 


Time-Truncated  Data 

The  EEC  CERT  test  data  is  from  a  time-truncated  test  truncated  at  50,044  hours.  To 
establish  the  AMSAA  model  parameters,  using  the  maximum  likelihood  estimation-,  from 
MIL-HDBK-189, 


A 


P  = 


N 


N  log  (T)  -  Z  log(ti  ) 

N-l 


A 

-,n  = 


N 


Where  N  =  the  total  number  of  failures,  T  -  the  cumulative  test  time,  and  tj  is  the  cumulative 
test  time  at  the  ith  failure. 

A  /A  * 

The  instantaneous  failure  rate  is  therefore  p  (T)  =  X  {3  T  P  “ 1 

A 

and  the  instantaneous  MTBF,  MT6F  =  1/£(T). 

A  A 

In  the  First  20,000  hours  of  CERT  test  data,  P  and  X  would  be  calculated  as  follows: 


Incident 

Cum  Time 

Log  (Cum  Time) 

1 

4.0 

1.386 

2 

20.8 

3.035 

3 

294.0 

5.684 

4 

315.0 

5.753 

5 

2459.0 

7.808 

6 

3100.0 

8.039 

7 

7274.0 

8.892 

8 

8996.3 

9.105 

9 

15522.1 

9.650 

10 

19988.7 

9.903 

69.255 

Final  Test  Total  Time  20,000  hours. 

10 _ 10_  _  _„r 

•'  10  (9.903)  -  69.189  “  29.846  ~ 

X  =  10  +  (20,000)0,335  =  0.3623 

p  (20,000)  =  0.335  X  0.3623  X  (20.000)0-335"1  =  1.675  X  10"4 

A 

MTBF  =  5970 
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Failure-Truncated  Data 


Occasionally  tests  are  stopped  after  a  prespecified  number  of  failures  have  occurred,  or 
when  a  failure  occurs,  a  decision  is  made  to  stop  the  test  until  the  problem  is  resolved.  These  are 
failure-truncated  tests.  Maximum  likelihood  estimates  for  p  and  X  are  as  follows: 


A 

P 


_ N _ 

(N-l)  log  (tN)  -  X  log  (t) 

N-l 


N;'  A 

,n  = 

i-l 


N 

T* 


If  the  CERT  test  were  failure  truncated  at  the  10th  failure,  estimates  of  P  and  X  would  be  as 
follows: 


P  “  9(9.903)-(69. 189-9.903)  "  ‘335 
k  “  (19988)'335  “  ‘362 

which  are  the  same  as  time-truncated  estimates  since  the  last  failure  occurred  so  close  to  the 
20,000  hour  point. 


Grouped  Data 

If  the  exact  time  at  which  failures  occur  is  not  known,  the  parameters  can  be  estimated 
from  the  following  equations: 


Where  N;  failures  have  occurred  in  the  interval  t; ,  t^  and  K  is  the  number  of  intervals, 
shown  in  the  following  equation: 


As  an  example,  consider  the  failure  experience  on  the  F100  engine  EEC  JFC-90 
accumulated  over  a  4  year  period  of  development.  Here  the  failures  and  time  are  recorded  by 
month,  listed  in  Table  D-4  and  shown  in  Figures  D-20,  D-21  and  D-22.  These  plots  allow  the 
analyst  to  see  the  data  in  Table  D-3  pictorially  as  the  AMSAA  model  sees  it.  Figure  D-20  shows 
cumulative  failures  vs  cumulative  time.  As  Duane  model  views  it,  Figure  D-22  shows  cumulative 
MTBF  vs  cumulative  time  and  Figure  D-21  shows  the  data  in  failure  rate  form. 


Field  Data 

It  may  be  appropriate  to  model  field  data  in  the  same  manner  as  development,  provided  the 
system  reliability  can  be  assumed  exponential  and  that  a  program  exists  to  address  new  failures 
as  they  occur  and  thus  improve  system  reliability.  In  most  cases,  field  data  will  be  grouped  by 
month  or  by  quarter. 
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TABLE  D-4.  EEC  JFC-90  DEVELOPMENT  DATA 


Tine 

Interval 

Cum  Incidents 

0 

78 

6 

78 

425 

15 

425 

1,105 

.9 

1,105 

2,439 

21 

2,439 

4,776 

27 

4.776 

7,998 

31 

7,988 

11,369 

34 

11,359 

15,263 

36 

14,263 

17,036 

37 

17,036 

20,000 

39 

Using  the  above  iterntivs 
8  =  0.294  and  X  ~  2.11 

techniques, 

Figure  D-20.  EEC  JFC-90  Development  Data  (Cumulative  Failures) 


As  an  example,  consider  the  F100  engine  control  field  experience  over  the  past  2  years, 
Figures  D-16, 17  and  18. 

£=  0.836,  ft  =  0.011. 


Regression  Estimators/Graphical  Methods 


Least  Squares  Regression  (LSR)  estimates  can  be  obtained  by  fitting  the  following  Model: 


log  (N)  =  log  (X)  +  p  log  (T\ 


104 


T.i  .-  .s ,-  -•L.-  j  tl  ■_£* J J_£> ^.-3 J-s j.-> ■'. --> .■■"-J-’V 


““¥SG 

uVm 


Average 
Failure  5 
Rate 


Cumulative  Test  Time  (Hours  x  1000) 


FD  270279 


Figure  D-21.  EEC  JFC-90  Development  Data  (Average  Failure  Rate) 
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Figure  D-22.  EEC  JFC-90  Development  Data  (Cumulative  MTBF) 

LSR  estimates  are  useful  in  getting  the  best  fit  and  in  getting  starting  values  for  the  iterative 
procedure  necessary  to  find  maximum  likelihood  estimators  with  grouped  data.  However,  Least 
Squares  Estimates  are  not  recommended  for  use  in  making  inference  about  the  current  or  future 
reliability. 

Estimates  can  be  obtained  graphically  as  indicated  in  MIL-HDBK-189,  page  133,  but  these 
estimates  can  also  be  unreliable,  and  hence  are  not  recommended. 
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c)  Goodne$$  of  FH  Tout 


A  statistically  valid  goodness  of  fit  test  should  be  applied  to  the  data  to  substantiate  that 
the  model  is  appropriate.  If  the  model  and  data  look  comparable  graphically  and  a  goodness  of  fit 
test  indicates  the  model  is  appropriate,  then  inferences  drawn  from  the  model  are  based  on  a 
sound  foundation. 

Continuously  Tracked  Data 

For  time-truncated  data  and  failure-truncated  data,  the  Cramer  Von  Mises  statistic  (C2)  is 
used  to  test  the  model  goodness  of  fit.  The  following  are  needed  to  calculate  this  statistic:  An 
unbiased  estimate  of  the  growth  parameter  (fi)  is  used  along  with  the  times  at  each  failure  (tj), 
the  total  time  (T)  and  the  total  number  of  failures  (N)  to  calculate  this  statistic. 

—  A 

P  =  — jq —  p  For  time-truncated  data 


-  M_2  A 

P  =  — jq—  P  For  failure-truncated  data 


For  time-truncated  data,  M  =  N,  for  failure-truncated  data  M  =  N  -  1.  If  the  calculated 
value  of  C:M  is  less  than  the  table  value  (at  a  prespecified  significance  level,  0.1,  0.05,  etc.)  then 
the  model  is  accepted. 

For  example,  with  the  EEC  CERT  data,  before  splitting  it  into  two  groups: 

P  =  0.623,  N  =  29 

«■*  OQ 

P  «  -g-  -0.623  =  0.602 

C|,  =  0.522. 

The  table  value  listed  in  Table  D-5  is  0.172  for  a  =  0.1;  therefore,  as  was  suspected  by 
looking  at  the  graphs  in  Figure  D-7,  the  model  does  not  fit.  After  breaking  the  data  at 
20,000  hours,  models  are  fit  to  each  new  phase  as  follows: 


A 

■*u 

=  0.335 

A 

d2 

=  0.922 

% 

=  0.302 

d2 

=  0.874 

r2 

10 

=  .039 

c219 

=  0.1167. 

Both  critical  values  (0.167  and  0.171)  are  not  less  than  the  calculated  values;  therefore,  modeling 
the  data  in  two  separate  parts  is  appropriate. 


This  statistic  (%2t  has  K-2  degrees  of  freedom.  A  table  value  for  this  statistic  can  be  found  by 
setting  the  significance  level,  a  =  0.1,  0.05,  etc.,  and  knowing  the  degrees  of  freedom  (K-2).  If  the 
calculated  value  of  a  is  less  than  the  table  value  x2(y  -  2,  l  -  a)  *  accept  the  model. 

Example  of  F100  electronic  engine  control  development  experience  is  listed  in  Table  D-6. 
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TABLE  D-6.  FlOO  EEC  DEVELOPMENT  EXPERIENCE 


Time 

Interval 

Cum  Incident* 

Ni 

ei 

-  e  if  /ei 

0 

78 

6 

6 

7.6 

0.335 

78 

425 

15 

9 

4.9 

3.430 

425 

1105 

19 

4 

4.1 

0.002 

1105 

2439 

21 

2 

4.3 

1.230 

2439 

4776 

27 

3 

4.6 

0.536 

4776 

7998 

31 

4 

4.1 

0.002 

7988 

11359 

34 

3 

3.3 

0.027 

11359 

14263 

36 

2 

2.2 

0.018 

14263 

17036 

37 

1 

1.9 

0.426 

17036 

20000 

39 

2 

1.8 

0.022 

II 

6.04 

A 

Uiing  the  above  iterative  techniquea,  6  -  0.294  and  k 

-  2.11 

The  calculated  value  (6.04)  is  less  than  the  table  value  (at  a  =  0.9)  (13.4)  of  Table  D-6;  therefore, 
the  model  is  accepted. 

Possible  Causes  for  Not  Passing  the  Goodness  of  Fit  Test 

If  the  AMSAA  model  fails  the  goodness  of  fit  test,  the  following  courses  of  action  are 
recommended: 


( 1 )  The  problem  could  be  bad  data.  Re-evaluate  the  preliminary  analysis. 

(2)  Major  shifts  in  the  program,  reliability  or  technology  level  may  have  gone 
unnoticed.  Recheck  the  test  phases  to  see  if  they  apply  (example  with  the 
EEC  CERT). 

(3)  It  may  be  the  case  that  the  model  just  doesn’t  fit  the  data.  If  the  data  plots 
a  smooth  curve  on  log-log  paper,  or  a  snake  like  curve  with  no  clear/distinct 
point  to  break  the  data,  then  the  AMSAA  model  may  not  be  appropriate, 
graphically  shown  in  Figure  D-23. 


The  problem  may  be  that  the  process  is  not  a  nonhomogeneous  poisson  process  (NHPP). 

Some  possible  clues  that  the  process  is  not  a  NHPP  are  as  follows: 

•  Can  a  failure  occur  before  testing  begins?  NHPP  assumes  N(0)  =  0,  i.e.,  that 
no  failure  can  occur  before  time  0.  Units  with  shelf  life  may  violate  this 
assumption. 

•  Do  failures  experienced  appear  to  influence  the  probability  of  future 
failures?  NHPP  assumes  that  the  process  has  independent  increments. 

•  Have  simultaneous  failures  consistently  occurred?  NHPP  assumes  that 
simultaneous  failures  are  impossible. 


Cumulative 

Incidence 


Y. 


Interval  estimates  for  p  can  be  obtained  for  time-truncated  data  as  follows: 

•  Find  the  tabled  value  of  Chi-Square  with  2N  degrees  of  freedom.  x22n,  a/2  an<* 

X22N,  1  -  0/2  Table  D-7). 

„  A 

Lower  Bound  =  X  2N,o/2  *  P/2N 
Upper  Bound  =  x22n.  1  -  a/2  •  S/2N 

•  For  failure-truncated  data. 

9  A 

Lower  Bound  =  x  2n  -  2,  a/2  *  P/2N 
Upper  Bound  =  x22n  -  2.1  -  a/2  *  1^/2  N 

A 

•  Consider  the  example  of  the  first  20,000  hours  of  CERT  data.  N  =  10,  P  = 

0.336  at  the  90%  confidence  level  (a  =  0.1). 

X220.o.05  =  10-9 

X220.0.95  =  31.4 

Lower  Bound  =  10.9  *  (  **")  **  0-18 
Upper  Bound  =  31.4  *  (  )  =  0.53 

Therefore,  based  on  the  AMSAA  model,  growth  is  occurring  with  90%  confidence.  The  true 
growth  parameter  is  between  0.18  and  0.53  with  90%  confidence. 

Testing  for  Growth  with  Grouped  Data 

There  is  a  Chi-Square  test  for  trend  that  can  be  applied  when  only  grouped  data  is 
a\  ailable.  The  Chi-Square  statistic  (x2)  is  computed  and  compared  to  the  table  value,  X2k-i,  i-a  > 
where  1-a  is  the  confidence  level  (90%).  If  the  calculated  value  is  greater  than  the  table  value, 
and  the  calculated  value  for  P,  (p)  is  less  than  1.0,  growth  can  be  substantiated. 

To  calculate  the  Chi-Square  value,  use  the  following  formula: 

2  K  (Nj_  NP.)2 

X  =  2- 

1- 1 


NP, 


Where  K  is  the  number  of  intervals  (each  interval  should  be  large  enough  so  that  N  Pj  >5),  N  is 
the  number  of  failures  in  interval  i,  P;  =  tj/  T,  tj  is  the  time  at  the  upper  limit  of  interval  i, 
T  =  Itj,N  =  INj. 
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TABLE  D-7.  PERCENTAGE  POINTS,  CHI-SQUARE  DISTRIBUTION 
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For  example,  consider  the  F100  EEC  field  experience,  shown  in  Table  D-8. 

TABLE  D-8.  EEC  JFC-90  FIELD  DATA 

(Ni  -  NP / 


Time 

Ni 

NP> 

0-200 

239 

0.11 

200-400 

159 

23.97 

400-600 

215 

1.62 

600-800 

256 

2.10 

800-1000 

303 

20.43 

1000-1200 

228 

0.15 

1200-1400 

253 

1.56 

1400-1600 

218 

1.08 

TB7T 

51.76  *= 

N  =  I  N, 

-  1871 

d  =  t./T  “  200/1600  -  0.125 

X27j  g  =  12.0,  so  reject  that  the  data  has  a  constant  failure  rate.  However,  two  intervals  1)  200-400 
and  2)  800-1000  appear  to  be  extremes.  Before  growth  is  concluded,  these  two  intervals  should  be 
thoroughly  researched. 

e)  Estimate*  of  Reliability 

Results  of  the  statistical  analysis  should  include  an  assessment  of  the  current  reliability  and 
a  projection  of  future  reliability. 

Estimate  of  Current  Reliability 

It  is  important  to  note  that  the  observed  reliability  at  a  particular  point  in  time  (T)  is  only 
an  estimate  of  the  true  reliability.  For  example,  in  the  EEC  CERT  test,  at  20,000  hours,  it  is 
possible  to  have  obtained  a  different  set  of  failures  had  s  different  set  of  hardware  been  tested. 

To  estimate  the  current  instantaneous  MTBF,  (M(t)) 


A  i  1  A 


In  the  CERT  test  example, 


mw  -  -w-w*20'000)""”  -  6026- 


To  estimate  the  instantaneous  failure  rate. 


An  interval  estimate  of  the  instantaneous  MTBF  (M(t))  can  be  obtained  for  time-  and  iailure- 
truncated  data  as  follows: 


•  Time-truncated  data,  look  up  values  of  L  and  U  in  Table  D-9  for  N  failures 
and  confidence  y. 

A 

Lower  Bound  =  M(t)  •  L 
A 

Upper  Bound  =  M(t)  •  U 

•  For  failure-truncated  data,  look  up  values  of  L  and  U  in  Table  D-10  for  N 
failures  and  confidence  y. 

To  illustrate  the  use  of  confidence  bounds  on  the  instantaneous  MTBF,  consider  the  EEC 
CERT  data,  M(t)  =  6026  N  =  10,  at  the  90%  confidence  level,  L  =  0.476  and  U  =  2.575. 
Therefore,  the  true  reliability  would  be  expected  to  be  between  2868  hours  and  15,517  hours  with 
90%  confidence  as  follows: 

0.476*6026  =  2,868 
2.575*6026  =  15,517. 

This  reliability  demonstrated  over  the  next  5000  hours  was  considerably  less  than  the  lower 
bound  (5000/5  =  1000  hr  MTBF).  Engineering  analysis  suggests  that  this  was  a  result  of 
continued  acceleration  of  unfixed  failure  modes  from  the  first  20,000  hour  interval. 

Estimates  of  Future  Reliability 

To  compute  an  estimate  of  the  reliability  at  a  future  time  T,  evaluate  M(t)  at  T  as  follows: 

/),  ,  1  1  'rii  -  ^ 

M(t)  =  -$■  ■  -if  ■  r 

This  prediction  assumes  that  the  reliability  growth  continues  as  in  the  past.  Historically, 
this  estimate  has  not  been  reliable  beyond  1  to  2  years  in  advance. 

For  long  term  predictions,  or  predicting  beyond  two  years),  use  the  planned  curve  to 
forecast  reliability.  The  expected  cumulat  ive  failure  (C(t))  rate  is  given  by  the  function: 

C(t)  =  yT“ 

M(t)  =  ayT®-1. 

The  assumption  made  in  this  projection  is  that  the  program  obtains  its  planned  growth. 

To  project  reliability  within  the  third  phase  of  the  EEC  103-2  development  program, 
assume  the  growth  rate  continues  at  the  same  rate  as  the  last  phase  of  CERT. 

A  A  iq 

(j  =  0.929.X  =- - -  =  1.415x10' 3 

H  (30,000f9“ 


M(T)  =  1  A-T1  - - - —  (33,000)'  '  0922  =  1225.0 

Md)  ff  1  0.922  •  1.415  •  10  ‘  3  v  ' 
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TABLE  D-9.  CONFIDENCE  INTERVALS  FOR  MTBF  FROM  TIME 
TERMINATED  TEST* 


IB 

.80 

L 

U 

.90 

L 

U 

.95 

L 

U 

.98 

L 

U 

2 

.261 

18.66 

.200 

38.66 

.159 

78.66 

.124 

198.7 

3 

.333 

6.326 

.263 

9.736 

.217 

14.55 

.174 

24.10 

4 

.385 

4.243 

.312 

5.947 

.262 

8.093 

.215 

11.81 

S 

.426 

3.386 

.352 

4.517 

.300 

5.862 

.250 

8.043 

6 

.459 

2.915 

.385 

3.764 

.331 

4.738 

.280 

6.254 

7 

.487 

2.616 

.412 

3.298 

.358 

4.061 

.305 

5.216 

8 

.511 

2.407 

.436 

2.981 

.382 

3.609 

.328 

4.539 

9 

.531 

2.254 

.4S7 

2.750 

.403 

3.285 

.349 

4.064 

10 

.549 

2.136 

.476 

2.575 

.421 

3.042 

.367 

3.712 

11 

.565 

2.041 

.492 

2.436 

.438 

2.852 

.384 

3.441 

12 

.579 

1.965 

.507 

2.324 

.453 

2.699 

.399 

3.226 

13 

.592 

1.901 

.521 

2.232 

.467 

2.574 

.413 

3.050 

14 

.604 

1.846 

.533 

2.153 

.480 

2.469 

.426 

2.904 

15 

.614 

1.800 

.545 

2.087 

.492 

2.379 

.438 

2.781 

16 

.624 

1.759 

.556 

2.029 

.503 

2.302 

.449 

2.675 

17 

.633 

1.723 

.565 

1.978 

.513 

2.235 

.460 

2.584 

18 

.642 

1.692 

.575 

1.933 

.S23 

2.176 

.470 

2.503 

19 

.650 

1.663 

.583 

1.893 

.532 

2.123 

.479 

2.432 

20 

.657 

1.638 

.591 

1.858 

.540 

2.076 

.488 

2.369 

21 

.664 

1.615 

.599 

1.825 

.548 

2.034 

.496 

2.313 

22 

.670 

1.594 

.606 

1.796 

.556 

1.996 

.504 

2.261 

23 

.676 

1.574 

.613 

1.769 

.563 

1.961 

.511 

2.215 

24 

.682 

1.557 

.619 

1.745 

.570 

1.929 

.518 

2.173 

25 

.687 

1.540 

.625 

1,722 

.576 

1.900 

.525 

2.134 

26 

.692 

1.525 

.631 

1.701 

.582 

1.873 

.531 

2.098 

27 

.697 

1.511 

.636 

1.682 

.588 

1.848 

537 

2.068 

28 

.702 

1.498 

.641 

1.664 

.594 

1.825 

.543 

2.035 

29 

.706 

1.486 

.646 

1.647 

.599 

1.803 

.549 

2.006 

30 

.711 

1.475 

.651 

1.631 

.604 

1.783 

.554 

1.980 

1C 

.729 

1.427 

.672 

1.565 

.627 

1.699 

•  573 

1.870 

40 

.745 

1.390 

.690 

1.515 

.646 

1.635 

.599 

1.788 

45 

.758 

1.361 

.705 

1.476 

.662 

1.585 

.617 

1.723 

50 

.769 

1.337 

.718 

1.443 

.670 

1.544 

.632 

1.671 

60 

.787 

1.300 

.739 

1.393 

.700 

1.481 

.657 

1.591 

70 

.801 

1.272 

.756 

1.356 

.718 

1.435 

.678 

1.533 

80 

.813 

1.251 

.769 

1.328 

.734 

1.399 

.695 

1.488 

LOO  ' 

.831 

1.219 

.791 

1.286 

.758 

1.347 

.722 

1.423 

For  N  >  100, 


L  -  (1  +  Z0,5  +  r/2/V2N)'2  U  =  (1  +  Z06 1  y/2/V2N)-2 
in  which  Z05+y/2  is  the  (0.5  +  y/2)-th  percentile  of  the  standard  normal  distribution. 


Taken  from  MIL-HDBK-189. 


f)  Estimates  of  Jumps  In  Reliability 

Occasionally,  it  is  necessary  to  estimate  the  amount  of  improvement  in  reliability  as  a  result 
of  several  delayed  fixes.  This  situation  usually  occurs  at  the  end  of  a  test  phase,  but  can  occur 
elsewhere.  The  following  assumptions  are  necessary  to  estimate  the  jumps: 

•  The  system  reliability  at  time  T,  (R(T)),  is  a  result  of  two  categories  of 
failure  modes:  A  ~  modes  which  are  not  fixed  and  B  ~  modes  for  which 
delayed  fixes  are  available. 

•  Each  failure  mode  in  the  system  has  a  constant  failure  rate,  Xj. 

^-A  =  £  =  £ 

Note  that  the  reliability  before  fixes  are  made  is 
r(o)  =  XA  +  XB. 

•  For  each  failure  mode  (Xj),  the  effectiveness  of  the  fix  (dj)  is  a  known 
constant.  (If  a  fix  is  known  to  reduce  the  failure  rate  by  1/2,  d  =  0.5). 

•  The  average  rate  at  which  a  new  problem  is  discovered  (H(t))  (not  old 
problems  reoccurring)  follow  a  NHPP  model  of  the  same  form  as  the 
AMSAA  model,  as  follows: 

h(t)  =  aPT*5-1. 


To  estimate  the  reliability  after  the  fixes  are  incorporated,  first  estimate  P 
and  a,  as  follows: 

a  m 

p  _  -  where  M  is  the  number  of  failure  modes, 

I  iog  (T/t.) 

i-1 

T  is  the  cumulative  time  on  test  (this  phase),  and  t;  is  the  cumulative  test 
time  at  the  ith  failure. 


M  T 

- 


Using  d,  compute  h(T)  =  -jr-- 


Compute  the  average  of  the  effectiveness  factors, 

A  1  f  , 

**d  =  M  '  rj di- 

Compute  the  term  B(T)  =  pj  h(T). 

If  Na  is  the  number  of  observed  failures  from  type  A  modes,  Nj  is  the  number 
of  observed  failures  from  the  ith  type  B  mode,  the  new  reliability  can  be 
computed  as  follows: 

i  M 

r(T)  =  (NA  +  Z(l-dON,)  +  B(t) 

1  i-i 

and  the  new  MTBF  =  i/r(T). 
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To  estimate  the  reliability  jump  as  a  result  of  delayed  fixes  from  the  EEC  103-2  CERT 
{Phase  2)  to  the  next  phase,  break  the  failure  data  into  failure  modes,  assign  an  effectiveness 
factor  to  fixes  and  estimate  the  current  failure  rate,  as  shown  in  Table  D-l  1. 

A  10 
P  ”  24.33  =  0,41 

■»  Q  A 

P  -  75-  ■  P  -  0.37 

h<T)  -  -  0-074/1000 

pd  =  0.692 

B(T)  -  *  h(T)  =  0.05113/1000 

r(T)  =  -50^00-  (3+5.25)  +  0.05113/1000  =  0.000216 

MTBF  =  4627 

Therefore,  an  MTBF  of  approximately  4600  hours  is  expected  through  the  next  phase. 

TABLE  D-ll.  ESTIMATE  OF  CURRENT  FAILURE  RATE 


Modes 

Mode 

Type 

Failure 

Rate 

N, 

d, 

Cum  Time 
of  Failure 

T/t, 

1 

B 

1/10000 

5 

0.9 

2.0 

25000.0 

2 

A 

1/50000 

1 

— 

20.8 

2404.0 

3 

A 

1/50000 

1 

— 

7274.0 

6.87 

4 

B 

14/50000 

14 

0.75 

8996.3 

5.56 

5 

B 

1/25000 

2 

0.9 

19988.7 

2.501 

6 

B 

1/25000 

2 

0.75 

20751.8 

2.410 

7 

B 

1/50000 

1 

0.30 

35342.5 

1.415 

8 

A 

1/50000 

1 

— 

36082.7 

1.386 

9 

B 

1/50000 

1 

0.75 

42586.2 

1.174 

10 

B 

1/50000 

1 

0.5 

43059.4 

1.161 

E.  INTERPRETATION  OF  RESULTS 

1.  Assessment 

The  point  estimate  of  the  instantaneous  MTBF  is  the  best  estimate  of  the  current  MTBF. 
Upper  and  lower  bounds  on  the  MTBF  show  the  possible  variation  which  could  have  occurred 
with  different  hardware. 

2.  Prediction 

Near  term  forecast  can  be  made  by  extrapolating  up  the  AMSAA  model  growth  curve  (the 
model  fit  within  each  test  phase).  Long  term  forecast  can  be  made  by  projecting  along  the  ideal  or 
planned  curve. 


r  -  »  u  *  w  - 
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3.  Control 


In  determining  whether  the  program  is  on  schedule,  the  planned  reliability  (considering  the 
contribution  of  the  present  test  phase)  can  be  compared  to  the  demonstrated  reliability  from 
assessment.  If  the  interval  estimate  of  MTBF  covers  the  planned  MTBF,  the  program  is 
progressing  within  reasonable  limits. 


APPENDIX  E 


TIME  SERIES  ANALYSIS 

A.  INTRODUCTION 

Many  parametric  and  non-parametric  methods  for  estimating  the  failure  rate  function  have 
been  developed.  One  disadvantage  in  these  methods  is  the  inflexibility  built  in  by  the  assumed 
model  and  a  lack  of  theory  for  forecasting. 

The  time  series  approach  is  free  of  any  assumptions  regarding  either  the  failure  distribution 
or  the  general  form  of  its  failure  rate  function.  This  approach  is  based  on  the  concept  that  since 
the  failure-rate  function  is  a  function  of  time,  the  estimated  failure  rates  are  constructed  as  being 
generated  by  a  time  series  process.  A  Box-Jenkins  time  series  model  will  be  used  to  approximate 
the  failure-rate  process  and  to  set  confidence  limits  on  these  forecasts. 

1.  Motivation  for  Time  Series 

Suppose  that  [t,]"_  ,  is  a  specified  sequence  on  the  time  axis,  and  let  Xc(t)  be  the  estimated 
cumulative  failure  rate  function  for  the  interval  [tj,  t(  ,  ,),  where  tj  <  t  <  t;  4  ,.  If  A.c(t)  depends 
only  on  observations  in  the  interval  (tj,  t;  +  ,),  and  those  observations  prior  to  tj,  then  [  Xc(t); 
t  >  0  ]  is  by  definition  a  time  series.  If  lc(t)  depended  upon  observations  at  time  ti+,  or  beyond  t; 
4  j,  then  [  A.(.(t) :  t  >  0  ]  would  not  be  a  time  series.  If  the  time  series  (  A.c(t) :  t  >  0  ]  can  be  made 
stationary  (remain  in  equilibrium  about  a  constant  mean  level)  by  various  methods  of  Box- 
Jenkins,  then  the  methods  of  Box-Jenkins  can  be  used  to  forecast  future  failure  rates. 

2.  Time  Series  Models  and  Notation 

Let  \.(t),  X(.(t  -  1),  Xc(t  -  2), ...  be  the  realizations  of  a  stochastic  process  at  equispaced 
time  points  t,  t  -  1, . . .  which  are  generated  by  a  sequence  of  independent  identically  distributed 
random  variables  at,  at  _  ,, . . .  with  mean  0  and  variance  ol .  This  sequence  of  random  variables  is 
sometimes  called  a  white  noise  sequence.  The  symbols  at  will  be  used  for  this  type  of  random 
variable.  The  following  sketch  illustrates  how  the  time  series  is  generated. 

White 

Noise 

a,  Linear  X(.(t) 

Filter 

The  white  noise  process  at  is  transformed  to  the  process  ^,.(t)  by  a  linear  filter.  This  linear 
filtering  operation  takes  a  weighted  sum  of  previous  white  noise  random  variables  to  form  Xc(t), 
as  shown  in  the  following  equation: 

\.(t)  =  p  +  at  +  t|/,  at  _  ,  +  \|f2at  _ . . .,  where  p  is  the  mean  of  the  series.  The  sequence  vg,, 
ML. . . .  form  what  is  called  the  linear  operator  that  transforms  a,  into  A.c(t). 

a)  General  Form  of  the  Time  Series  Model 

A  suitable  model  that  describes  many  of  the  commonly  occurring  nonstationary  time  series 
process  is  the  AutoRegressive  Integrated  Moving  Average  process  of  order  p,  d,  q  (ARIMA 
(p,  d,  q)).  To  illustrate  the  general  form  of  the  model,  if  w|'(t)  is  the  dth  difference  of  Xc(t),  (i.e.,  for 

. d-l^t)  =  \.(t)  -  Xc(t  -  1);  for  d  -  2,  wf(t)  -  w.’(t)  -  w('(t  -  1)  =  *.c(t)  -  \(t  -  1)  -  (Xc(t  -  1)  - 

\.(t  -  2))"-  Xc(t  -  1)  +  X(.(t  --  2);  etc.),  the  form  of  the  model  is  given  in  equation  (1), 

w?(t)  =  <P,  wj  (t  -  1)  +  <p2  w?  (t  -  2)  <n,  w?  (t  -  p)  f  a,  -  0  ,  a,  ,  —  02  a,  0  a, 

*  71  \ 
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where  ip,,  <p2,  .  .  .  <pp  are  the  p  unknown  autogressive  parameters,  and  0P  02,  .  .  .  0q  are  the  q 
unknown  moving  average  parameters.  Equation  (1)  can  be  simplified  using  the  backward  shift 
operator  B  defined  by  B*  •  Xc(t)  =  Xc{t  -  k)  where  k  >  1.  For  instance,  B1  •  Xc(t)  =  Xc(t  -  1).  We 
can  rewrite  the  general  ARIMA  (p,  d,  q)  model  given  equation  (1)  in  terms  of  this  backward 
notation  given  in  equation  (2): 

<p(BK(t)-0(BK 

(«/ 

where 

tp(B)  =-  1  -  q>,  B1  -  <p2  B2 . . .  -  <pp  BP  and 

0(B)  =  1  -  0,  B1  -  02  B2 . . .  -  0q  Bq 

A  further  generalization  of  the  ARIMA  (p,  d,  q)  given  in  equation  (2)  is  to  include  a  deterministic 
polynomial  trend,  so  that  the  model  now  becomes  equation  (3): 

q,(BK(t)=*0„  +  0(Bfc. 

(3) 

b)  Specific  Cases  of  the  General  Model 

Many  of  the  parameters  of  the  general  ARIMA  (p,  d,  q)  model  will  not  be  needed.  If  d  =  0, 
then  w;.'(t)  =  A,(t)  and  the  process  is  known  as  an  autoregressive  moving  average  (ARIMA  (p,  0,  q)). 
The  form  of  this  model  is  given  in  equation  (4). 

<p(B)  A.c(t)  =  0(B)  at  or 

W  ~  <Pi  M1  “  1)  ~  <P2  *  c(t  "  2)  •  •  •  “  «Pp  Vt  -  p)  =  at  -  0!  at  _  , .  . .  -  0  at 

(4) 

If  P  =  0,  the  time  series  is  known  as  an  integrated  moving  average  (ARIMA  (0,  d,  q)),  whose  form 
is  given  in  equation  (5): 


w?(t)  =  0(B)at  or 
w;!(t)  =  a,  -  0,a,  ,...  -0„a, 


If  p  -  0  and  d  =  0,  the  time  series  is  known  as  a  moving  average  (ARIMA  (0, 0,  q)),  whose  form  is 
given  in  equation  (6). 

w"(t)  =  X,(t)  =  0{B'a,  or 

Mt)  =  a,  -  0,a,  ,...  -0„a,  , 

(6) 

If  q  =  0  and  d  =  0,  the  time  series  is  known  as  an  autoregressive  process  (ARIMA  (p,  0,  0)).  The 
form  of  this  model  is  given  in  equation  (7). 

<p(B)w;'(t)  =  <p(B)A,(t)  =  a,  or 

Mt)  -  <P|A.,(t  -  1)  -  <P*Mt  ~  2)...  -  <P,X(t  -  p)  =  a, 


I 


I'M* 
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c)  Model  Constraints 


The  time  series  to  be  modeled  must  satisfy  two  requirements:  invertibility  and  stationarity. 
To  en  ure  invertibility  and  stationarity  of  the  process,  the  parameters  <p j,  <p2, . . .  <pp  and  0lf  02,  •  •  • 
0q  will  need  to  fulfill  certain  requirements.  The  process  of  invertibility  will  now  be  explained  by 
use  of  an  example. 

Consider  the  moving  average  process  of  order  1  (ARIMA  (0,  0, 1):  Xc(t)  =  at  -  0,  at  _  i  = 
(1  -  O,  B)  at.  This  moving  average  can  be  expressed  as  a  weighted  sum  of  previous  failure  rates, 

and  a  random  element  at,  Xc(t)  =  at  -  0,  Xc(t  —  1)  —  0f  X,.(t  -  2)  -  0,'  Xc(t  -  3) - From  the  above 

equation,  if  I  0, 1  >  l,the  weights  associated  with  the  Xc(t  -  i),  i  =  1,  2 . increases  as  i 

increases,  which  says  that  the  process  gives  more  emphasis  to  the  oi., ‘nervations  which  have 
occurred  more  and  more  into  the  past.  The  requirement  that  i  0,  i  <  1  is  made,  and  the  series  is 
defined  to  lie  invertible.  In  general,  it  is  a  requirement  that  the  roots  of  0(B)  =  0  be  less  than  one 
in  absolute  value  for  the  process  to  be  invertible.  In  the  same  manner,  the  time  series  will  be 
stationary  if  the  roots  of  <p(B)  =  0  are  less  than  one  in  absolute  value. 

d.  The  Recommended  Time  Series  Model 

From  theory  and  practice,  an  integrated  moving  average  process  of  order  1,  (ARIMA 
(0, 1, 1)),  is  a  satisfactory  model  for  functions  of  the  failure  rate  process.  In  practice,  unless  there 
is  reason  to  believe  the  existence  of  a  deterministic  component,  0„  in  equation  (3)  is  set  equal  to 
zero.  Another  satisfactory  time  series  model  is  an  ARIMA  (0, 1,  0)  model,  A.c(t)  -  A,c(t  -  1)  =  at. 
Although  the  ARIMA  (0,  1,  0)  process  fits  the  time  series  well,  this  model  does  not  aid  in  the 
forecasting  of  future  rates. 

Characteristics  of  the  Recommended  Model 

The  form  of  the  ARIMA  (0, 1, 1)  model  used  for  the  prediction  of  the  Xc(t)’s  is  written  below 
in  three  equivalent  forms. 

=  \(t  -  1)  +  a,  -  0a,  ,  (g) 

(1  -  B)  ^(t)  =  (1  -  0B]a„  since  =  ^(t  -  1) 


<p(B)  X,(t)  =  ©(Bja,,  where  <p(B)  =  1  -  B  and  0(B)  -  (1  -  0B).  (1Q) 

The  latter  form,  equation  (10),  will  be  used  for  purposes  of  simplicity.  0  is  a  parameter  to  be 
estimated,  and  it  may  be  recalled  that  1 0 1  <  1  for  purposes  of  invertibility. 

For  purposes  of  forecasting,  it  will  be  necessary  to  express  the  ARIMA  (0, 1, 1)  model  as  a 
linear  function  of  the  independent  identically  distributed  random  variables  as  follows: 

Mt)  =  a,  +  V|/,a,  ,+V2a,  (n) 

It  can  be  easily  verified  that  the  vj/;  =  1  —  0,  i  =  1,2, ...  for  the  ARIMA  (0,  1,  1)  model  so 
that  equation  (11)  now  becomes: 


v|/,(t)  =  a,  +  (1  -  0)a,  ,  +  (1  -  0)a,  ,  +  ... 


(12) 
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The  ARIMA  (0, 1, 1)  model  may  also  be  expressed  as  an  infinite  weighted  sum  of  previous 
failure  rates,  Xc(t),  plus  a  random  component  at: 

Ml)  -  KiM*  -  1)  +  n-A(t  -  2)  +  ...  +  a,  (13) 

It  can  be  verified  that  x,  =  1  -  0,  *3  =  0(1  -  0),  x3  =  02(1  -  0),  ...  or  in  general  it}  = 
0* "  *(1  -  0),  so  that  equation  (13)  now  becomes: 

Xt.(t)  =  (1  -  0)  Xc(t  -  1)  +  0(1  -  0)  Xc(t  -2)  +  02  (1  -  0)  Xc(t  -  3)  +  .  .  .  +  at 


Notice  in  the  above  equation,  the  failure  rate  at  time  t,  Xc(t),  depends  less  and  less  on  past 
observations.  Also,  as  0  approaches  one,  the  weights  die  out  more  slowly. 

3.  Forecasting 

Future  failure  rates  will  be  predicted  using  the  ARIMA  (0, 1, 1)  model.  Suppose  at  time  t,  a 
future  failure  rate  at  time  t  +  8,  Xc(t  +  8),  8  >  1,  shall  be  forecasted.  At  time  t  +  8,  the  ARIMA 
(0, 1, 1)  model  may  be  written  as: 


Ml  +  8)  =  X,(t  +  8  -  1)  +  a,  , ,  -  0a,  , ,  ,. 

(15) 

Box-Jenkins  shows  that  the  minimum  mean  square  error  forecast  at  origin  t,  for  lead  time  8  ,  is 
the  conditional  expectation  of  A.r(t  +  8)  at  time  t.  These  conditional  expectations  at  origin  t  are: 


M*  +  1)  -  Ml)  -  0a, 


Mt  +  8)  =  Mt  +  e  -  i),e>2. 


Hence,  for  all  lead  times,  the  forecasts  at  origin  t,  will  follow  a  straight  line  parallel  to  the  time 


The  variance  of  the  lead  8  forecasts  is  needed  so  that  confidence  limits  can  be  established. 
The  variance  of  these  forecasts  at  time  origin  t  is  given  by  equation  (17): 

Var(Mt  +  8))  =  of  ■  (1  +  (8  -  1X1  -  0f) ,  where  of  is  the  variance  of  the  at’s. 


Notice  that  the  variance  for  the  first  forecast  is  of,  and  this  variance  increases  as  the  forecast 
leads  are  increased. 


a)  Model  Identification 

The  first  and  most  obvious  way  to  start  the  process  of  model  identification  is  to  plot  the 
cumulative  failure  rate,  Xc(t),  as  a  function  of  time.  If  the  failure  rates  do  not  seem  to  be  centered 
about  a  fixed  mean,  the  data  is  probably  nonstationary,  and  therefore  the  failure  rates  (A.c(t)’s) 
should  be  differenced  to  produce  the  desired  state  of  stationarity.  Examination  of  the 
autocorrelations  will  also  provide  insight  as  to  whether  or  not  the  time  series  is  stationary  or 
nonstationary.  The  autocorrelation  function  will  now  be  defined. 
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Given  a  finite  time  series  of  cumulative  failure  i-ates,  Xc(l),  Xc(2),  .  .  .  XC(N)  of  N 
observations,  the  autocovariance  at  lag  k,  yk,  is  defined  to  be: 


yk  =  covariance  (X.(t),  Xc(t  +  k)]. 
The  autocorrelation  at  lag  k,  pk,  is  defined  to  be 


yk  will  be  estimated  by  Ck,  where 

c, - -jj-i* m  -  xm# + k)  -  m 

t-1 


(18) 


(19) 


(20) 


where  Xc(t)  is  the  mean  of  the  cumulative  failure  rates. 
pk  will  be  estimated  by  rk,  where 

Cv 

rk  =  T? 


(21) 


For  a  stationary  time  series,  the  estimated  autocorrelation  function,  (rk),  for  all  lags  should 
die  out  fairly  rapidly.  If  the  estimated  autocorrelations  do  not  die  out  rapidly,  the  time  series 
should  be  differenced  to  obtain  stationarity.  The  partial  autocorrelation  function  will  also 
provide  insight  as  to  whether  the  time  series  should  be  differenced.  The  idea  of  partial 
autocorrelation  will  be  explained  by  an  example.  The  simple  correlation  between  two  failure  rates 
A.c(t)  and  X.c(t  -  2)  can  perhaps  be  equal  to  zero,  while  the  correlation  between  Xc(t)  and  \c(t  -  2) 
adjusted  for  Xc(t  -  1)  may  not  be  zero.  Hence,  the  partial  au„jcorrelation  between  Xc(t)  and  X(t  - 
2),  can  be  looked  upon  as  the  correlation  between  Xc(t)  and  Xc(t  -  2)  adjusted  for  X,c(t  -  1).  If  the 
partial  autocorrelation  at  lag  1  is  close  to  unity,  while  the  partial  autocorrelations  at  lags  greater 
than  one  are  close  to  zero,  this  is  evidence  that  the  series  should  be  differenced.  Finally,  the 
LJUNG  modification  of  the  Box-Pierce  Q  statistic  produces  a  test  whose  distribution  is  Chi- 
square  (x2)  under  the  null  hypothesis  that  the  series  is  white  noise.  If  this  statistic  is  not 
statistically  significant,  and  thus  the  process  can  be  labeled  as  a  white  noise  process,  no 
differencing  should  be  made,  and  the  process  can  be  modeled  as  Xc(t)  =  at.  However,  if  this  test  is 
not  statistically  significant,  we  can  conclude  that  the  series  should  be  further  modeled.  From  our 
suggested  model,  the  cumulative  failure  rates  should  be  differenced  once,  A.c(t)  -  A.c(t  -  1), 
although  the  above  characteristics  should  be  examined  carefully  to  reach  this  conclusion. 

Sometimes,  the  mean  of  the  differenced  series  is  not  equal  to  "ero,  and  an  extra  parameter 
0„,  called  a  trend  parameter  should  be  included  in  the  model.  Usually  we  shall  assume  that  this 
mean  is  equal  to  zero,  or  equivalently  9„  -  0,  unless  it  is  evident  from  the  data  that  the  problem 
has  a  nonzero  mean.  To  statistically  conclude  if  the  mean  of  the  differenced  series  is  equal  to 
zero,  compute 


iw'(t) 

t-2 

w'.(t)  =  ^  — j-  where  we  recall  that  w,'  =  K  (t)  -  K{t  - 1) 


and  compare  it  with  its  approximate  standard  error  given  in  equation  (23). 


°K(t))  = 


(  C„(l  +  2r,)  )"2 


(22) 


(23) 


V 


After  deciding  that  the  series  should  be  differenced  once  and  a  test  for  a  trend  parameter 
has  been  made,  examination  of  the  autocorrelations  and  partial  autocorrelations  for  the 
differenced  3eries,  Xc(t)  -  Xc(t  -  1),  should  be  made.  For  the  suggested  ARIMA  (0, 1, 1)  model, 
the  autocorrelations  should  drop  off  after  the  first  lag,  and  the  partial  autocorrelations  should 
tail  off  exponentially.  To  ensure  that  the  residuals  are  white  noise,  the  LJUNG  modification  of 
the  Box-Pierce  Q  statistic  should  be  performed  again,  after  the  ARIMA  (0, 1, 1)  model  has  been 
identified.  This  statistic  is  calculated  on  the  model  residuals  to  test  the  hypothesis  that  they  are 
white  noise. 


Only  one  of  the  above  possible  solutions  can  satisfy  the  invertibility  condition  that  -  1  <  0  <  1. 
Estimating  p,  by  r,„  and  substituting  into  the  above  equation  will  yield  a  preliminary  estimate  for 
0.  Nonlinear  estimation  will  improve  upon  this  estimate,  until  a  suitable  solution  has  been  found. 


The  final  parameter  to  be  estimated  is  the  variance  of  the  at’s,  which  is  a;.  An  estimate  of  a\ 
may  be  obtained  by  substituting  the  preliminary  estimate  of  0  and  y0  by  its  estimate  C0  in  to 
equation  (27)  and  solving  for  a"' 


v,  =  a-(.  +  02) 

B.  PROCEDURE  FOR  IDENTIFYING  TIME  SERIES  MODELS 


(27) 


The  F100  electronic  engine  control  (EEC)  data  generated  over  4  years  of  development  will 
be  used  as  an  illustrative  example.  The  model  form  and  parameters  will  be  derived  using  the  time 
series  methods  in  Box-Jenkins.  The  computer  software  used  to  evaluate  the  data  is  the 
Statistical  Analysis  System  (SAS  Institute  Inc.,  Cary,  North  Carolina). 

1.  Model  Identification 


a)  Differencing  the  Time  Series 

The  first  step  in  trying  to  identify  the  form  of  the  model  is  to  plot  the  original  data  of  the 
cumulative  failure  rates  against  time  beginning  at  the  first  failure.  The  EEC  data  does  not  seem 
to  be  fluctuating  about  a  fixed  level,  as  shown  in  Figure  E-l.  This  is  one  indication  that  the  series 
should  be  differenced  to  remove  the  changing  shifts  in  level.  This  is  one  type  of  nonstationary 
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time  series.  Another  indication  that  the  series  should  be  differenced  is  to  calculate  the  estimated 
autocorrelations  and  estimated  partial  autocorrelations  of  the  undifferenced  series.  For  the  EEC 
data,  the  estimated  autocorrelations  follow  a  gentle  linear  decline,  thus  suggesting  that  the  series 
is  nonstationary,  as  opposed  to  a  rapid  geometric  decline  of  the  autocorrelations  for  a  stationary 
series  (see  Computer  Output  1).  The  estimated  partial  autocorrelation  shows  a  correlation 
approaching  one  at  the  first  lag,  and  a  correlation  near  zero  at  all  lags  greater  than  one  (see 
Computer  Output  2).  This  indicates  an  autoregressive  model  with  the  parameter  9  approaching 
one,  again  indicating  that  the  series  should  be  differenced.  For  the  undifferenced  EEC  data,  the 
autocorrelation  check  for  white  noise  is  significant  so  that  the  hypothesis  that  the  series  is  white 
noise  is  rejected,  (see  Computer  Output  3).  Hence,  there  is  strong  evidence  that  the  process  is 
nonstationary. 


Figure  E-l.  Cumulative  Failure  Rate  of  Undifferenced  EEC  Data 


After  differencing  the  series  once  (wci(t)  =  Xc(t)  -  A,c(t  -  1)),  a  plot  of  this  first  difference 
should  be  made  to  indicate  whether  the  process  has  obtained  stationarity.  For  the  EEC  data,  the 
first  difference  seems  to  have  made  the  series  stable  (as  graphically  shown  in  Figure  E-2). 
Calculation  of  the  estimated  autocorrelations  and  estimated  partial  autocorrelations  should 
further  aid  in  this  decision.  The  autocorrelations  for  the  one-differenced  EEC  data  are  close  to 
zero  for  all  lags  greater  than  zero,  as  opposed  to  the  linear  decline  of  the  autocorrelations  for  the 
undifferenced  series  (see  Computer  Output  4).  Notice  also  that  the  partial  autocorrelations  for 
the  first  lag  is  not  close  to  one,  as  opposed  to  the  partial  autocorrelations  approachii,  z  one  for  the 
first  lag  of  the  undifferenced  series  (see  Computer  Output  5).  Hence,  there  is  strong  evidence  that 
the  series  has  obtained  stationarity  by  differencing  the  series  once. 


The  variance  estimate  of  the  series  a'l  has  decreased  dramatically  by  the  differencing 
procedure  (0.000964022  to  0.000162855)  as  expected  (see  Computer  Output  6  and  Computer 
Output  7,  respectfully).  Overdifferencing  the  series  will  increase  this  variance.  As  an  example, 
suppose  the  EEC  data  had  been  differenced  twice  (w*(t)).  Notice  that  the  variance  of  the  series 
increased  from  0.000162855  to  0.000265891,  indicating  another  reason  that  the  series  should  not 
be  second  differenced  (see  Computer  Output  7  and  Computer  Output  8,  respectfully). 


Cumulative 

Failure 

Rate 

Per  1000 


FD  270284 

Figure  E-2.  Cumulative  Failure  Rate  of  First  Differenced  EEC  Data 

b )  Examination  of  the  Autocorrelation  Functions 

Now  that  the  series  has  obtained  stationary  through  first  differencing,  further  examina¬ 
tion  of  the  autocorrelation  and  partial  autocorrelation  functions  should  be  done.  The  Statistical 
Analysis  System  (SAS)  computes  these  functions,  as  well  as  an  inverse  autocorrelation  function, 
whose  behavior  is  somewhat  similar  to  the  partial  autocorrelation  function.  Table  E-l  is  a  chart 
of  the  behavior  of  the  estimated  functions  after  stationarity  has  been  obtained. 

TABLE  E-l.  ESTIMATED  FUNCTIONS  BEHAVIOR  AFTER  STATION¬ 
ARITY  HAS  BEEN  OBTAINED 


Autocorrelation 

Function 

Partial  Autocorrelation 
Function 

Inverse  Autocorrelation 
Function 


Autoregressive 

Moving  Avernge  Autoregressive  Moving  Average  White 

of  Order  Q _ of  Order  i' _ of  Order  P,  Q  Noise 

I)(Q)  T  TO 


D(Q)  means  the  function  drops  off  to  zero  after  lag  Q. 

D(P)  means  the  function  drops  off  to  zero  after  lag  P. 

T  means  the  function  tails  of  exponentially. 

0 _ means  the  function  is  zero  at  all  nonzero  lags. 


Examining  the  estimated  autocorrelations  for  the  once-differenced  EEC  data  reveals  a 
slight  drop  off  to  7.ero  after  lag  one,  although  the  autocorrelation  does  in  fact  lie  within  two 
standard  errors  of  zero  (see  Computer  Output  4).  The  estimated  inverse  autocorrelations  and 
estimated  partial  autocorrelations  show  a  slight  decline  towards  zero,  beginning  with  the  first  lag 
(see  Computer  Output  5).  These  statistics  indicate  that  the  differenced  series  could  be  a  moving 
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average  of  order  1  (Xc(t)  -  Xc(t  -  1)  =  -  Ga,^  _  j)  or  a  white  noise  sequence 

(Xc(t)  -  Xc(  t  -  1)  =  at).  The  autocorrelation  check  for  white  noise  indicates  that  the  hypothesis 
that  the  differenced  series  is  a  white  noise  process  cannot  be  rejected  (see  Computer  Output  9). 
However,  to  obtain  information  to  aid  the  process  of  forecasting,  a  moving  average  parameter 
will  be  added  to  the  statistical  model.  After  adding  this  extra  term  to  the  once-differenced  EEC 
data,  a  significant  moving  average  parameter  was  obtained;  and,  in  adding  this  extra  term,  the 
residuals  are  statistically  white  noise  by  observing  the  autocorrelation  check  of  residuals  (see 
Computer  Output  10).  The  estimate  for  this  moving  average  parameter  is  0.225118  (see 
Computer  Output  10).  A  trend  parameter  could  be  necessary  in  the  model.  Testing  for  the 
significance  of  this  parameter  yields  a  nonsignificant  t  statistic  (-0.33).  Thus,  a  trend  parameter 
is  unnecessary  in  the  time  series  model  (see  Computer  Output  11 ).  The  final  time  series  model  is 
Xc(t)  -  y t  -  1)  =  at  -  0.225118  at  _  ,  with  =  0.000158384. 

2.  Forecasting 

For  the  forecasted  value  for  January  of  the  sixth  year,  using  the  forecasting  equation 
yt  +  1)  -  X(.(t)  -  0a(,  at  t  =  51,  the  forecasted  value  is  X(.(51)  =  0.0020  (see  Computer  Output  12 
and  13).  Consequently,  since  X(.(t  +  8)  =  X,.(t  +  2  -  1)  for  8  >  2,  X(.(51)  =  X,,(52)  =  . . .  =  0.0020. 
The  predicted  cumulative  failure  rate  is  a  straight  line  parallel  to  the  time  axis.  The  variance  of 
these  forecasts  increase  with  time  according  to  the  equation,  Var  (X,.(  t  +  8))  =  al  (1  +  (2  -  1)  (1  - 
9)~).  Thus,  the  variance  for  Xc(51)  =  a'2„  =  0.000158384,  and  the  standard  deviation  is  0.0126.  This 
variance  increases  as  the  forecast  in  time  increase  (see  Computer  Output  12  and  13).  The  1  year 
predicted  cumulative  failure  rate  is  a  line  parallel  to  the  time  axis,  as  shown  in  Figure  E-3),  and 
no  cumulative  failure  rate  decline  is  predicted. 


Figure  E-3.  Cumulative  Failure  Rate  of  EEC  Data  With  Prediction 
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COMPUTER  OUTPUT  1  THROUGH  13 


Computer  Output  1 

TIME  SERIES  ANALYSIS  OF  EEC  DATA  16:29  TUESDAY,  OCTOBER  9,  1983 

ARIMA  PROCEDURE 

NAME  Or  VARIABLE  =  RATE 

MEAN  OF  WORKING  SERIES=  0.0267S15 
STANDARD  DEVIATION  =  0.0310487 
NUMBER  OF  OBSERVATIONS:  51 
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Computer  Output  2 

TIME  SERIES  ANALYSIS  OF  EEC  DATA  16=29  TUESDAY,  OCTOBER  4,  1963 
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Computer  Output  3 


TIME  SERIES  ANALYSIS  OF  EEC  DATA 


14:31  THURSDAY,  OCTOBER  6,  1983 


AUTOCORRELATION  CHECK  FOR  WHITE  NOISE 


TO 

CHI 

AUTOCORRELATIONS 

LAG 

SQUARE 

OF 

PROS 

6 

196.13 

6 

0.000 

0.911 

0.843 

0.792  0.728 

0.659  0.582 

12 

240.64 

12 

0.000 

0.504 

0.427 

0.339  0.283 

0.220  0.147 

18 

242.12 

18 

0.000 

0.C92 

0.043 

0.014  -0.013 

-0.043  -0.077 

24 

257.00 

24 

0.000 

-0.103 

-0.126 

-0.149  -0.171 

-0.191  -0.209 

Computer  Output  4 


TIME  SERIES  ANALYSIS  OF  EEC  DATA  16:29  TUESDAY,  OCTOBER  4,  1983 

ARIMA  PROCEDURE 


NAME  OF  VARIABLE  =  RATE 
PERIODS  OF  DIFFERENCING3  1. 

MEAN  OF  WORKING  SERIES3- .00059741 
STANDARD  DEVIATION  =  0.0127615 
NUMBER  OF  CODER VAT IONS3  50 


AUTOCORRELATIONS 


LAG  COVARIANCE 
0  .000162055 

CORRELATION 

1.00000 

-1 

1 

9  8  7  6  5  4  3 

2101234567891  STD 

|«XXXXXXXXXXXXXX**XX*|  0 

1  -3.625E-05 

-0.22256 

1 

xxxx| 

1  0.141421 

2  -4.692E-06 

-0.028S1 

1 

*1 

1  0.148261 

3  7. 350E-06 

0.04513 

1 

1  * 

1  0.148373 

4  4 . C4CE-06 

0.C24S6 

1 

I 

!  0.140647 

5  6.854E-C6 

0.04209 

1 

i# 

1  0.14273 

6  4.C57E-07 

0.00249 

1 

1 

1  0.14896S 

7  -2.017E-06 

-0.01238 

1 

1 

I  0.148969 

0  .CC0010717 

0.06581 

1 

|» 

1  0.14399 

9  -3.3C5E-05 

-0.20294 

1 

xxxxj 

1  0.14957 

10  0.00000339 

0.05152 

1 

1# 

!  0.154979 

11  9.875E-C6 

0.06064 

1 

l« 

1  0.155322 

12  -1.934E-05 

-0.11074 

1 

XX  j 

1  0.155794 

13  -9. 19SE-06 

-0.05649 

1 

*1 

I  0.157594 

14  -1.012E-05 

■0.06215 

1 

*1 

1  0.157993 

IS  -0. 723C-07 

-0.00536 

1 

1 

!  0.153437 

16  -8.3C0E-07 

-0.0C515 

1 

1 

1  0.15049 

17  -e.ie:e-06 

-0.01323 

1 

1 

1  0.153494 

1-3  -2 . 33CE-C6 

-0.01432 

1 

! 

1  0.150516 

19  -2  .OI-4E-06 

-0.01759 

1 

1 

1  0 . 150542 

20  -4.C52E-C3 

-0 .00030 

1 

1 

1  0.150501 

21  -1. 0532-06 

-0.CC646 

1 

1 

1  0.150531 

22  -1.609E-06 

-0.009C8 

1 

1 

1  0.150506 

23  -1.849E-06 

-0.01135 

1 

1 

1  0.150393 

24  -2 . 194E-06 

-0.01347 

1 

1 

1 

MARKS  TWO  STANDARD  ERRORS 

1  0.150614 

TCjCUKV  'f~~*r*ur+ 


Computer  Output  6 


TIME  SERIES  ANALYSIS  OF  EEC  DATA 

ARIMA:  PRELIMINARY  ESTIMATION 

WHITE  NOISE  VARIANCE  EST=. 000964022 

ARIMA:  LEAST  SQUARES  ESTIMATION 

ITERATION  SSE  LAM3DA 
0  0.085663  1.0E-05 

1  0.035663  1.0E-06 

PARAMETER  ESTIMATE  STD  ERROR  T  RATIO  LAG 


VARIANCE  ESTIMATE  =0.00167967 
STO  ERROR  ESTIMATE  =  0.0609337 
NUMBER  Or  RESIDUALS:  51 

CORRELATIONS  OF  THE  ESTIMATES 


AUTOCORRELATION  CHECK  OF  RESIDUALS 


14:31  THURSDAY,  OCTOBER  6,  1983 


TO 

CHI 

AUTOCORRELATIONS 

LAG 

SQUARE 

DF 

PROS 

6 

220.99 

6 

O.OCO 

0.946 

0.833 

0.831 

0.776 

0.718 

0.655 

12 

301.29 

12 

0.000 

0.592 

0.530 

0.463 

0.414 

0.368 

0.315 

1C 

321.93 

13 

0.000 

0.272 

0.240 

0.217 

0.197 

0.178 

0.160 

24 

329.36 

24 

0.000 

0.145 

0.132 

0.120 

0.108 

0.093 

0.090 

n 


m 

M 

I 


m 

■•1**1 


Computer  Output  7 

TIME  SERIES  ANALYSIS  OF  EEC  DATA  14:31  THURSDAY,  OCTOBER  6,  1983  9 

ARIMA:  PRELIMINARY  ESTIMATION 

WHITE  NOISE  VARIANCE  EST=. 000162855 

ARIMA:  LEAST  SQUARES  ESTIMATION 

ITERATION  SSE  LAMBDA 
0  0.0081606  1.0E-05 

1  0.0081606  1.0E-06 


PARAMETER  ESTIMATE  STD  ERROR  T  RATIO  LAG 


VARIANCE  ESTIMATE  =.000163212 
STD  ERROR  ESTIMATE  =  0.0127754 
NUMBER  OF  RESIDUALS1  50 


CORRELATIONS  OF  THE  ESTIMATES 


AUTOCORRELATION  CHECK  OF  RESIDUALS 


TO 

CHI 

AUTOCORRELATIONS 

LAG 

SQUARE 

DF 

PR03 

6 

2.79 

6 

0 .  G35 

-0.214  -0.022  0.051  0.031 

0.048  0.009 

12 

6.94 

12 

0.862 

-0.0C6  0.071  -0.197  0.056 

0.065  -0.114 

10 

7.41 

13 

0.9S6 

-0.053  -0.059  -0.003  -0.003 

-0.011  -0.012 

24 

7.46 

24 

0.999 

-0.016  0.001  -0.005  -0.008 

-0.010  -0.012 

Computer  Output  8 


TIME  SERIES  ANALYSIS  OF  EEC  DATA  14:29  THURSDAY,  OCTOBER  6,  1983  13 

ARIMA:  PRELIMINARY  ESTIMATION 

WHITE  NOISE  VARIANCE  EST=. 000265891 

ARIMA:  LEAST  SQUARES  ESTIMATION 

ITERATION  SSE  LAMBDA 
0  0.0131646  1.0E-05 

1  0.0131646  1.0E-06 

PARAMETER  ESTIMATE  STD  ERROR  T  RATIO  LAG 


VARIANCE  ESTIMATE  =.000268665 
STD  ERROR  ESTIMATE  =  0.016391 
HUMBER  OF  RESIDUALS=  49 

CORRELATIONS  OF  THE  ESTIMATES 


AUTOCORRELATION  CHECK  OF  RESIDUALS 


TO 

CHI 

AUTOCORRELATIONS 

LAG 

SQUARE 

OF 

PR  03 

6 

2.58 

6 

0.059 

-0.209 

-0.055  0.029 

-0.023 

0.035  0.001 

12 

8.94 

12 

0.708 

-0.050 

0.194  -0.201 

0.006 

0.135  -0.052 

18 

9.09 

18 

0.958 

-0.002 

-0.046  -0.000 

0.005 

0.001  -0.000 

24 

9.11 

24 

0.997 

-0.013 

0.005  0.002 

0.002 

0.000  -0.009 

135 


Computer  Output  10 


TIME  SERIES  ANALYSIS  OF  EEC  DATA  16^29  TUESDAY,  OCTOBER  4,  1983  14 

ARIMA:  PRELIMINARY  ESTIMATION 

MOVING  AVERAGE  ESTIMATES 
1  0.22256 


WHITE  NOISE  VARIANCE  EST= . 000155169 

ARIMA:  LEAST  SQUARES  ESTIMATION 


ITERATION  SSE 

MAI  ,1 

LAMBDA 

0 

0.00776009 

0.222562 

1.0E-05 

1 

0.00776084 

0.225274 

1.0E-06 

2 

0.00776084 

0.225118 

1.0E-07 

PARAMETER 

ESTIMATE 

STD  ERROR 

T  RATIO 

LAG 

MAI  ,1 

0.225118 

0.139147 

1.62 

1 

VARIANCE  ESTIMATE  =.000158384 
STD  ERROR  ESTIMATE  =  0.0I2S85I 
NUMBER  OF  RESIOUALS=  50 

CORRELATIONS  OF  THE  ESTIMATES 

MAl.l 

MAI, I  1. 000 


AUTOCORRELATION  CHECK  OF  RESIDUALS 


TO 

LAG 

CHI 

SQUARE 

OF 

FROB 

6 

0.69 

5 

0.903 

-0.002 

12 

4.16 

11 

0.965 

0.006 

18 

5.40 

17 

0.996 

-0.093 

24 

5.60 

23 

1.000 

-0.021 

AUTOCORRELATIONS 

-0.010  0.062  0.058  0.066  0.024 

0.033  -0.1S4  0.026  0.044  -0.126 

-0.085  -0.024  -0.012  -0.018  -0.021 
-0.005  -0.009  -0.014  -0.017  -0.017 
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Computer  Output  11 

TIME  SERIES  ANALYSIS  OF  EEC  DATA  16:29  TUESDAY,  OCTOBER  4,  1983  18 

ARIMA:  PRELIMINARY  ESTIMATION 

CONSTANT  TERM  ESTIMATE^. 00059741 
WHITE  NOISE  VARIANCE  EST= . 000162855 

ARIMA:  LEAST  SQUARES  ESTIMATION 

ITERATION  SSE  MU  CONSTANT  LAMBDA 

0  0.00814275  -6.0E-04  -.00059741  1.0E-05 

1  0.00314275  -6.0E-04  -.00059741  1.0E-06 

PARAMETER  ESTIMATE  STO  ERROR  T  RATIO  LAG 

MU  -.00059741  0.00182306  -0.33  0 

CONSTANT  ESTIMATE  =-.00059741 


C 


/ 


VARIANCE  ESTIMATE  =.000166179 
STD  ERRCR  ESTIMATE  =  0.012891 

NUMBER  OF  RCSIOUALS=  50 

CORRELATIONS  OF  THE  EST  rMATES 

MU 

MU  1.000 


AUTOCORRELATION  CHECK  OF  RESIDUALS 


TO 

CHI 

AUTOCORRELATIONS 

LAG 

SQUARE 

OF 

PRGB 

6 

2.92 

5 

0.712 

-0.223  -0.029  0.045  0.025 

0.042  0.002 

12 

7.20 

11 

0.783 

-0.012  0.066  -0.203  0.052 

0.061  -0.119 

18 

7.73 

17 

0.97?. 

-0.056  -0.062  -0.005  -0.005 

-0.013  -0.014 

24 

7.80 

23 

0.999 

-0.018  -0.000  -0.006  -0.010 

-0.011  -O.O'I 

i 


i 


i 
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Computer  Output  12 


TIME  SERIES  ANALYSIS  OF  EEC  DATA 
FORECASTS  FOR  VARIABLE  RATE 


16:29  TUESDAY,  OCTOBER  4,  1903  27 


OBS 

FORECAST 

STD  ERROR 

LOWER  95X 

UPPER  95X 

ACTUAL 

RESIDUAL 

2 

0.0318 

0.0126 

0.0072 

0.0565 

0.1134 

0.0815 

3 

0.0950 

0.0126 

0.0704 

0.1197 

0.0880 

-0.0070 

4 

0.0096 

0.01 26 

0.0649 

0.1143 

0.0833 

-0.0063 

5 

0.0847 

0.0126 

0.0601 

0.1094 

0.0833 

-0.0014 

6 

0.0837 

0.0126 

0.0590 

0.1083 

0.0833 

-0.0003 

7 

0.0834 

0.0126 

0.0587 

0.1081 

0.0833 

-0.0001 

8 

0.0833 

0.0126 

0.0587 

0.1080 

0.0806 

-0.0027 

9 

0.0813 

0.0126 

0.0566 

0.1059 

0.0768 

-0.0044 

10 

0.0778 

0.0126 

0.0532 

0.1025 

0.0763 

-0.0015 

11 

0.0767 

0.0126 

0.0520 

0.1013 

0.0562 

-0.0205 

12 

0.0608 

0.0126 

0.0361 

0.0855 

0.0593 

-0.0015 

13 

0.0596 

0.0126 

0.0350 

0.0843 

0.0594 

-0.0002 

14 

0.0595 

0.0126 

0.0348 

0.0341 

0.0439 

-0.0156 

15 

0 . 0474 

0.0126 

0.0227 

0.0720 

0.0353 

-0.0121 

16 

0.0380 

0.0126 

0.0133 

0.0627 

0.0279 

-0.0101 

17 

0.0302 

0.0126 

0.0055 

0.0549 

0.0262 

-0.0040 

13 

0.0271 

0.0126 

0.0024 

0.0518 

0.0244 

-0.0027 

19 

0.0250 

0.0126 

0.0004 

0.0497 

0.0219 

-0.0032 

20 

0.0226 

0.0126 

-0.0021 

0.0472 

0.0195 

-0.0030 

21 

0.0202 

0.0126 

-0.0045 

0.0449 

0.0172 

-0.0030 

22 

0.0179 

0.0126 

-0.0063 

0.0425 

0.0164 

-0.0015 

23 

0.0167 

0.0126 

-0.0079 

0.0414 

0.0149 

-0.0018 

24 

0.0153 

0.0126 

-0.0093 

0.0400 

0.0130 

-0.0023 

25 

0.0136 

0.0126 

-0.0111 

0.0382 

0.0112 

-0.0024 

26 

0.0117 

0.0126 

-0.0130 

0.0364 

0.0094 

-0.0023 

27 

0.0099 

0.0126 

-0.0148 

0.0346 

0.0086 

-0.0013 

23 

0.0039 

0.0126 

-0.0158 

0.0336 

0.0CV6 

-0.0013 

29 

0.0079 

0.0126 

-0.0167 

0.0326 

0.0075 

-0.0004 

30 

0.0076 

0.0126 

-0.0170 

0.0323 

0.0072 

-0.0004 

31 

0.0073 

0.0126 

-0.0174 

0.0320 

0.0063 

-0.0005 

32 

0.0069 

0.0126 

-0.0178 

0.0316 

0.0067 

-0.0002 

33 

0.0067 

0.0126 

-0.0179 

0.0314 

0.0060 

-0.0007 

34 

0.0062 

0.0126 

-0.0185 

0.0303 

0.0055 

-0.0006 

35 

0.0057 

0.0126 

-0.0190 

0.0303 

0.0050 

-0.0006 

36 

0.0052 

0.0126 

-0.0195 

0.0290 

0.0045 

-0.0007 

37 

0.0046 

0.0126 

-0.0200 

0.0293 

0.0041 

-0.0006 

38 

0.0042 

0.0126 

-0.0205 

0.0203 

0.0038 

-0.0003 

39 

0.0039 

0.0126 

-0.0207 

0.0206 

0.0035 

-0.0004 

40 

0.0036 

0.0126 

-0.0210 

0.0283 

0.0033 

-0.0003 

41 

0.0034 

0.0126 

-0.0213 

0.0200 

0.0031 

-0.0002 

42 

0.0032 

0.0126 

-0.0215 

0.0278 

0.0030 

-0.0002 

43 

0.0030 

0.0126 

-0.0217 

0.0277 

0.0030 

-0.0000 

44 

0.0030 

0.0126 

-0.0217 

0.0277 

0.0028 

-0.0002 

45 

0.0C28 

0.0126 

-0.0218 

0.0275 

0.0026 

-0.0002 

46 

0.0027 

C .0126 

-0.0220 

0.0273 

0.0025 

-0.0002 

47 

0.0025 

0.0126 

-0.0222 

0.0272 

0.0023 

-0.0002 

48 

0.0024 

0.0126 

-0.0223 

0.0270 

0.0022 

-0.0001 

49 

0.0023 

0.0126 

-0.0224 

0.0269 

0.0021 

-0.0001 

50 

0.0022 

0.0126 

-0.0225 

0.0268 

0.0021 

-0.0001 

51 

0.0021 

0.0126 

-0.0226 

0.0267 

0.0020 

-0.0001 

—  FORECAST 

BEGINS - 

— 

52 

0.0020 

0.0126 

-0.0227 

0.0267 

53 

0.0020 

0.0159 

-0.0292 

0.0332 

54 

0.0020 

0.0187 

-0.0346 

0.0306 

I 


4 
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Computer  Output  13 


TIME  SERIES  ANALYSIS  OF  EEC  DATA  16: 29  TUESDAY,  OCTOBER  4,  1983 

FORECASTS  FOR  VARIABLE  RATE 


OBS 

FORECAST 

STD  ERROR 

LOWER  95/J 

55 

0.0020 

0.0211 

-0.0393 

56 

0.0020 

0.0232 

-0.0435 

57 

0.0020 

0.0252 

-0.0473 

53 

0.0020 

0.0270 

-0.0509 

59 

0 .0020 

0.0207 

-0.0543 

60 

0.0020 

0.0303 

-0.0574 

61 

0.0020 

0.0318 

-0.0604 

62 

0.C020 

0.0333 

-0.0633 

63 

0.0020 

0.0347 

-0.0660 

UPPER  9S‘A  ACTUAL  RESIDUAL 
0.0433 
0.0475 
0.0513 
0.0549 
0.0583 
0.0614 
0.0644 
0.0673 
0.0700 
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